Race Condition and Deadlock Prevention: The Ownership Problem Behind Concurrency Bugs

What Race Condition and Deadlock Prevention Actually Protects

Concurrency correctness is a narrow slice of what code governance covers, but it's the slice that turns into the incident nobody can explain. A race condition rarely shows up as a clean stack trace. It shows up as a support ticket about a balance that's wrong by exactly one transaction, a report built from another customer's data, or a job that ran twice with no error anywhere in the logs, because nothing actually failed and two things just happened in the wrong order, the same ordering problem our distributed tracing guide covers for request flow instead of shared state.

None of the seven principles below are about developer carelessness. They're about the fact that concurrent execution is the default the moment a service handles more than one request at a time, and correctness under concurrency has to be designed in, not caught by testing. CWE-362, the formal name for a race condition, and CWE-833, the formal name for a deadlock, are exactly the gap discipline alone can't close.

1. State With No Named Owner Isn't Safe

A request handler stashes the current user in a module-level dictionary "just for this request." A date formatter gets promoted to a class-level field so it isn't rebuilt on every call. Under concurrent load, two requests interleave and one overwrites the other mid-flight. Never let request-scoped state live outside a local variable or an explicit context object, and never share an object that isn't documented as thread-safe across threads.

public class ReportService {
  private static final SimpleDateFormat FORMAT =
    new SimpleDateFormat("yyyy-MM-dd"); // shared, not thread-safe
}

public String formatDate(Date d) {
  return DateTimeFormatter.ISO_LOCAL_DATE
    .format(d.toInstant().atZone(ZoneId.systemDefault()));
}

Constructing an object per call costs some allocation overhead. If profiling shows that cost matters, the fix is a genuinely immutable type or a proper pool, not a shared mutable field with a comment promising to be careful.

2. Read-Then-Write Must Be Atomic, Not Careful

A balance check followed by a debit, or a getInstance() with a null check before constructing a singleton, reads a value, decides, then acts. Two threads can both pass the check before either acts: two debits against one balance, two singletons built. The fix is fusing the read and the write into one atomic step, never adding a second check.

if account.balance >= amount:
    account.balance -= amount
    db.save(account)

UPDATE accounts SET balance = balance - %s
WHERE id = %s AND balance >= %s

The double-checked locking idiom fails the same way for lazy initialization. The fix is a documented thread-safe construct, like an initialization-on-demand holder, never a second null check.

3. Lock Order Must Be Fixed, Not Incidental

A transfer function locks account A then B. A reverse transfer locks B then A. Neither is wrong alone, but if both run at once, each holds one lock and waits for the other, forever: the textbook deadlock. Whenever a path acquires more than one lock, it must use one fixed order derived from a stable key, never argument order.

Account first = a.id.compareTo(b.id) < 0 ? a : b;
synchronized (first) { synchronized (second) { ... } }

A quick diagnostic: grep every path that acquires two or more locks. If the order isn't derived from a stable key, a deadlock is waiting for the right interleaving.

This adds a small ordering step to every multi-lock code path. That cost is trivial compared to a production deadlock that only reproduces under load.

4. What You Do Inside a Lock Is the Real Risk

A cache-refresh function holds a lock while making a slow HTTP call. Every other thread waiting on that lock blocks for however long that call takes. Never perform I/O, a database query, or a message publish between acquiring and releasing a lock, and always release it through a construct that guarantees release on the failure path too.

data, err := http.Get(url) // outside the lock
mu.Lock()
defer mu.Unlock()
cache[key] = data

This sometimes means doing work twice: fetching data once outside the lock, then acquiring it only to apply the result, a small cost traded for never turning one slow dependency into full lock contention.

Every lock acquired manually and released "at the end of the function" never gets released when the function throws. Use try-with-resources, defer, or a context manager, every time.

5. A Lock That Can't Expire Isn't Safe

A thread acquires a lock and hangs. A downstream call never returns, and because the acquisition was unbounded, every other thread waiting on it now waits forever too. One stuck thread becomes a full outage. Every lock and semaphore acquisition needs an explicit timeout and a handled failure path, never an unbounded blocking wait.

A lock without a timeout isn't a lock. It's a single point of failure
waiting for one stuck thread to trigger it.

Mutual exclusion across service instances needs a distributed lock
with a bounded lease. An in-process mutex only ever protects threads
within one process.

When a lock is relied on for correctness rather than mere efficiency, occasional failure isn't acceptable; it has to hold every time, which is the standard a bounded, renewable lease is built to meet.

6. Concurrency Is a Budget, Not a Default

An async handler calls a synchronous database driver inside an event loop callback, and that one blocking call stalls every other request the loop was servicing. A fan-out job spawns one goroutine per row with no upper bound, and a batch that was fifty rows in staging becomes five hundred thousand in production.

Bound concurrency explicitly with a fixed-size executor, semaphore, or capped parallelism whenever fanning out over unbounded input, and submit work to a shared executor created once at startup, never one built inside a request handler.

Node.js serves many clients with a small number of threads, so a callback that blocks one of them keeps that thread from handling any other client for as long as it stays blocked.

7. Ordering Assumptions Don't Survive Team Boundaries

Team A's consumer assumes messages arrive in the order they were published, because that's how local testing behaved. Team B's producer, optimizing for throughput, fans out across partitions with no such guarantee. Both teams wrote correct code by their own definition, and the assumption was never written down anywhere both teams could see it.

Establish ordering explicitly with a synchronization primitive, a version comparison, or a queue that guarantees order, and treat that guarantee as a contract visible to every team that depends on it.

This is the version of the concurrency problem a single team's code review cannot catch. Two services can each be internally correct and still deadlock the moment their independent ordering assumptions meet in production.

How Pandorian Enforces This Guide

Every principle above survives a design review. Few survive six months of independent team decisions, because concurrency violations aren't visible from inside any one pull request; they're visible only under load. Pandorian enforces this guide across three surfaces:

Caught at PR level

A static field assigned from inside a request handler, a multi-lock acquisition with no stable ordering, a blocking call inside an async function: each is a checkable code shape, flagged with the line, the rule, and a fix before it merges.

Full scans

New code discipline doesn't retroactively fix the shared date formatter from three years ago or the lock acquired without a timeout in a module nobody has touched since. A full scan establishes the baseline across the whole codebase.

Scheduled scans

A dependency upgrade that changes a library's thread-safety guarantees, a refactor that moves a lock acquisition inside a new I/O call, a new consumer added with no concurrency budget. Scheduled scans catch drift PR checks, by construction, never see.

A Taste of the Guideline Package

Four of the checks that ship in this package. Each runs at the PR level, on full or scheduled scans, or several at once.

FAQ