Retry and Timeout Best Practices:
7 Rules for Resilient Calls

What Retry and Timeout Discipline Actually Prevents

Timeouts, retries, and circuit breakers are the three controls that decide whether a degraded dependency stays a degraded dependency or becomes an outage. They are usually treated as three independent settings on a client. They are one budget: the caller's deadline, divided among attempts, spent by whichever layer gets to it first.

The reason this goes wrong at scale is arithmetic. A gateway that retries three times, calling a service whose HTTP client retries three times, calling a driver that retries twice, produces eighteen attempts for one user request. Every layer's policy is defensible in isolation. The product is a self-inflicted denial of service delivered to a dependency that was already struggling. None of the seven principles below are about individual carelessness - they are about the fact that resilience is a property of the whole call chain, and the call chain is not owned by anyone.

1. Every Outbound Call Needs an Explicit Timeout

The incident does not look like a timeout problem. It looks like your service becoming unresponsive while its dependency is merely slow: threads accumulate in a wait state, the connection pool drains, healthy requests queue behind sick ones, and your latency graph goes vertical while the dependency's error rate stays flat. Set both a connection timeout and a request or read timeout on every HTTP client, gRPC channel, database client, cache client, broker client, and third-party SDK. Never inherit the library default - several of the most widely used ones are unbounded, which makes your service's default policy "wait forever."

resp = requests.post(f"{BILLING_URL}/charges", json=payload)
# no default timeout: this can hang for the life of the socket

resp = requests.post(
    f"{BILLING_URL}/charges",
    json=payload,
    timeout=(1.0, 4.0),  # (connect, read)
)

A good starting value is the downstream service's p99.9 latency plus a margin, the approach Amazon documents for choosing an acceptable rate of false timeouts. The tradeoff is real: a tight timeout converts slow successes into failures, and if your dependency's p50 sits close to its p99.9 you will spend engineering time tuning this rather than picking a number once.

2. A Deadline Travels With the Request

Without propagation, service C happily begins four seconds of work for a request service A abandoned two seconds ago. You pay full capacity for results nobody will read, and the wasted work is invisible on every dashboard because from C's perspective the request succeeded. Pass the caller's remaining deadline or cancellation context into every downstream call, and check it before starting further work.

ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
// discards whatever budget the original caller had left

resp, err := inventory.CheckStock(ctx, req)
// ctx carries the original deadline; gRPC sends the remainder
// on the wire as grpc-timeout

gRPC makes deadline propagation the default in Go and Java, and the gRPC team's guidance is unambiguous: always set one, and let it travel. In HTTP services the equivalent is a request-scoped context with an absolute deadline, threaded through rather than reconstructed at each hop. Diagnostic worth running: pick your slowest endpoint and name the service that will still be working on it thirty seconds after the browser gave up. If you cannot, deadlines are not propagating.

3. Retry Only Failures That Can Succeed Later

Retrying a 400, 401, 403, 404, 409, or 422 is not resilience; it is a loop that turns one client bug into sustained load against an endpoint that already gave you the answer. The signature in production is a rate-limit alert on an endpoint whose legitimate traffic has not changed. Restrict retries to timeouts, connection failures, and explicitly retryable statuses - 429, 502, 503, 504 - and give every policy a maximum attempt count.

const RETRYABLE = new Set([429, 502, 503, 504]);
for (let attempt = 0; attempt < 4; attempt++) {
  try { return await client.send(cmd); }
  catch (e) {
    if (!RETRYABLE.has(e.statusCode) && !isTransport(e)) throw e;
    await sleep(delayFor(attempt, e.headers?.["retry-after"]));
  }
}
throw new RetriesExhausted();

Unbounded retry loops, while-true with a sleep, and recursive retries with no depth limit all belong to one category: a policy with no upper bound on the load it can generate. When a server tells you when to come back, believe it - honour Retry-After or the provider's documented rate-limit reset header rather than substituting your own schedule.

The cost is that a genuinely transient 409 on a contended resource will now surface to the caller instead of resolving quietly. That is the correct trade: a conflict is a business outcome, and the caller is the layer that knows whether re-reading and re-submitting is safe.

4. Backoff Without Jitter Rebuilds the Herd

Every caller observed the failure at roughly the same time, so every caller retries at roughly the same time. The dependency recovers, absorbs a synchronized burst, and fails again - a recovery oscillation that looks like the dependency flapping when in fact your fleet is beating it back down. Compute delays with exponential growth plus randomized jitter, and cap the maximum delay.

const CAP_MS = 20_000, BASE_MS = 100;
const delayFor = (attempt) =>
  Math.random() * Math.min(CAP_MS, BASE_MS * 2 ** attempt);

AWS's exponential backoff and jitter analysis is the canonical treatment, and it shows full jitter outperforming both fixed and unjittered exponential schedules on total work and completion time. The same reasoning applies well beyond retries: scheduled jobs that all fire at the top of the hour are a herd nobody thinks of as one.

Rule of thumb: if you can predict the exact second your fleet's next retry wave will land, so can your dependency's saturation curve.

5. Non-Idempotent Retries Need an Idempotency Key

The nastiest version of this failure is invisible from the client side: the first attempt succeeded, the response was lost on the way back, the client saw a timeout and retried. Nothing in your logs says "double charge." The customer's statement does. Send an idempotency key derived from a stable business identifier on any operation with a non-reversible effect, and never retry such an operation without one.

// inside the retry loop
headers.set("Idempotency-Key", UUID.randomUUID().toString());

String key = "charge:" + order.getId() + ":" + order.getVersion();
headers.set("Idempotency-Key", key);  // stable across attempts

Stripe's idempotent requests model is the reference implementation worth copying: the server stores the status code and body of the first request under the key and replays it for every later attempt, including failures. If you are building the receiving end, this is a contract obligation rather than a client convenience - it belongs in the spec alongside the rest of your public API design standards.

6. Circuit Breakers Are Load Shedders, Not Health Checks

Timeouts and retries alone cannot stop a failing dependency from consuming your capacity: they bound each call, but you still pay the full timeout on every one, and under sustained failure that is enough to exhaust your own threads. Wrap each remote dependency in a breaker with an explicit failure threshold, open interval, and half-open trial policy - Martin Fowler's circuit breaker writeup remains the clearest statement of the three-state machine. Then do the part most implementations skip: define in code what the caller returns while the circuit is open.

A cached value, a queued write, a reduced response, a declared error: anything except an unhandled exception propagating to a user. Pair the breaker with a bounded pool or semaphore per dependency, so one saturated dependency cannot consume the execution capacity every other dependency shares. At the infrastructure layer, Envoy's outlier detection does the host-level version of the same job.

Breakers make testing harder and can extend recovery time - a breaker still open on a dependency that has already recovered is its own small outage. That is what the half-open trial is for, and why breaker state transitions belong in a metric rather than a log line.

7. Retry Ownership Belongs to One Layer

This is the principle that makes the previous six insufficient on their own. Every team implemented a defensible policy. The gateway retries because gateways should be forgiving. The service retries because the SDK made it a one-line config. The driver retries because that is its default. Eighteen attempts leave the building for one user request, and every review that approved a piece of it was correct.

Assign retry responsibility to exactly one layer per outbound dependency, configure the layers above and below it not to retry, and bound the total retry budget - the sum of attempt timeouts and backoff delays - below the timeout of the operation containing it. A budget that can exceed the caller's deadline is not a retry policy; it guarantees the caller times out before the last attempt lands, wasting every attempt after the first.

Diagnostic worth running: name the one layer that retries calls to your payment provider. If two people give different answers, you already have a multiplier in production. This is the same class of problem as microservice boundary discipline - locally correct, globally wrong, and visible only from a view of the whole codebase.

How Pandorian Enforces This Guide

Every principle above resolves to a shape in code: a client constructed without a timeout argument, a retry loop with no attempt ceiling, a fixed sleep where a jittered delay should be, an idempotency key generated inside the loop it is supposed to span. That makes them enforceable rather than aspirational.

Caught at PR level

A new HTTP client without an explicit timeout, a retry policy that catches a 4xx, or a fixed-interval sleep in a retry path surfaces with the line, the rule, and a suggested fix - while the author still has the context to act on it.

Full scans

An on-demand scan across the whole codebase sets the day-one baseline, and it is the only surface that sees cross-layer problems. A gateway retry policy and a service retry policy that are individually fine and jointly a 9x multiplier live in different repositories and never appear in the same diff.

Scheduled scans

The same full scan on a cadence, catching drift introduced outside any PR: an SDK upgrade whose default retry behaviour changed, a dependency added to a shared client factory, a breaker threshold tuned during an incident and never reverted.

A Taste of the Guideline Package

Four of the checks that ship in this package - each runs at the surface where it catches the problem earliest.

FAQ