Distributed Tracing Best Practices: 7 Rules for Context Propagation
Most organizations have tracing deployed and still cannot answer the question they bought it for. A trace that loses its context at hop four is not a partial trace - it is two unrelated traces, and neither one contains the answer.
TL;DR
- A request that begins without an established trace context produces spans that can never be joined to anything upstream of it.
- Context that is not injected into an outbound call ends there, and the downstream service starts a new trace it cannot know is a continuation.
- Queues and scheduled jobs are where traces die, because the boundary is a payload rather than a header and nothing propagates it by default.
- A proprietary correlation header buys you string matching in a log search; the standard headers buy you a causal graph.
- A trace identifier threaded through function signatures will be dropped by the first developer who adds a call path, which is every call path eventually.
- A span that only ends on the success path leaves the failure case - the one you are debugging - as a gap rather than an error.
- Trace quality is a fleet property: one service that drops context or floods cardinality degrades every trace passing through it, and other teams pay.
What Trace Context Propagation Actually Is
A trace context is a small piece of data - a trace identifier, the current span identifier, and sampling flags - that travels with a request across every boundary it crosses. Propagation is the act of extracting it on the way in and injecting it on the way out. Everything else in a tracing stack is downstream of that mechanism working.
The concept dates to Google's Dapper paper, and the modern wire format is the W3C Trace Context recommendation: a traceparent header carrying identity, and a tracestate header carrying vendor-specific data. OpenTelemetry uses it as the default, which means an instrumented service already speaks it - the failures are almost never in the SDK. They are at the seams. Tracing adoption plateaus because the value of a trace is superlinear in coverage: at 60% instrumentation you do not get 60% of the benefit, you get the ability to prove that something happened somewhere in the other 40%.
1. Every Entry Point Establishes a Trace Context
A request that begins without a trace context produces spans that cannot be joined to anything, which is indistinguishable in the backend from work that never happened. The gap is rarely at the front door - the API gateway is instrumented first and everyone assumes that covers it. It is the admin endpoint, the internal gRPC service, the webhook receiver, the cron entry point, the consumer added after the rollout.
export async function handleWebhook(req) {
const span = tracer.startSpan("webhook"); // no extract: new root
...
}const parent = propagation.extract(context.active(), req.headers);
await context.with(parent, () =>
tracer.startActiveSpan("webhook", async (span) => { ... })
);Extract the incoming trace context at every entry point, and generate a new one when none is present. The rule has to be "every entry point" rather than "every public entry point," because the internal ones are where multi-hop traces get stitched, and a break there splits one trace into two that no query can reunite. Diagnostic worth running: list every way work enters each service - HTTP routes, gRPC methods, queue consumers, cron entries, webhook handlers, admin tooling - then check which of those establishes a context.
2. Every Outbound Call Carries the Context Forward
Context that is not injected into an outbound call ends there, and the downstream service begins a fresh trace it has no way to know is a continuation. Auto-instrumentation covers the common HTTP and gRPC clients, which is why this failure concentrates where auto-instrumentation does not reach: a third-party SDK with its own transport, a hand-rolled client, a connection pool that predates the tracing rollout.
resp = httpx.post(PARTNER_URL, json=payload) # no propagator, no span
with tracer.start_as_current_span("partner.submit"):
headers = {}
inject(headers) # writes traceparent / tracestate
resp = httpx.post(PARTNER_URL, json=payload, headers=headers)Inject the current trace context into the headers of every outbound HTTP request, gRPC call, and third-party SDK call. OpenTelemetry's context propagation documentation describes the extract-inject cycle as the mechanism the whole model rests on. While you are there, create a child span for every database query, cache operation, message publish, and outbound service call - an untraced dependency call is a flat region in the waterfall where the latency you are hunting usually lives.
3. Queues and Jobs Are Where Traces Die
An asynchronous boundary carries no headers, so nothing propagates across it unless someone writes the code that does. This is the highest-yield principle in the guide, because the break is invisible from both sides: the producer's trace looks complete and correctly terminated at the publish, and the consumer's trace looks complete and correctly rooted at the handler. Neither team sees a problem. The person explaining why an order took nine minutes sees two traces and no causal link.
const carrier = {};
propagation.inject(context.active(), carrier);
await queue.publish({ body, headers: carrier });
// consumer side
const parent = propagation.extract(context.active(), msg.headers);
await context.with(parent, () => handle(msg));Inject the trace context into message headers or the job payload at publish or enqueue time, and extract and re-establish it at the start of the consumer or job handler. OpenTelemetry's messaging semantic conventions are explicit that consumer traces cannot be correlated with producer traces unless the creation context is attached to the message and propagated.
The same applies to scheduled work, batch runners, and anything that resumes processing from a durable record. Wherever a system deliberately crosses a durability boundary, it is also crossing a causality boundary, and only one of those two is handled by default.
4. Standard Headers Beat a Custom Correlation ID
A proprietary correlation header gives you string matching in a log search; the standard headers give you a causal graph with parentage and timing. Most organizations that built an X-Request-ID convention did so before tracing was a product decision, and it survives because it works well enough for the one query people run. It does not survive contact with a vendor boundary, a managed service, a service mesh, or a partner API - all of which speak traceparent and none of which have heard of your header.
inject(headers); // traceparent + tracestate
span.setAttribute("order.id", orderId); // correlation, not propagationPropagate trace context using traceparent and tracestate. Never define a proprietary correlation header as the primary propagation mechanism, and never propagate only an application-specific identifier. Keeping a business correlation ID as an additional attribute is fine and often useful; making it the mechanism means every future integration begins with a translation layer.
Migrating an established correlation-ID convention is real work, and for a period you will propagate both. That is the correct interim state - the mistake is treating the interim as the destination.
5. Context Lives in Ambient Scope, Not Signatures
A trace identifier threaded through function parameters will be dropped by the first developer who adds a call path, and every call path is eventually added by someone. The parameter-passing approach also produces its own tax: every intermediate function acquires an argument it does not use, which reviewers correctly identify as noise and periodically remove.
func Reserve(ctx context.Context, traceID string, orderID string) error // the next call path added will not pass traceID
func Reserve(ctx context.Context, orderID string) error // span and trace identity travel in ctx, automatically
Carry the trace context in the platform's ambient context mechanism - a context object, async-local storage, a thread-local scope. Then close the loop with logs: attach the current trace and span identifiers as structured fields on every log record where a context exists. That is what turns a log line into an entry point to the trace, and it builds directly on structured logging standards and log hygiene.
6. Spans Must Close on the Failure Path
A span that ends only on the success path leaves the failure case as a gap, which is the exact case you opened the trace to investigate. Worse, the orphaned span often stays open until a timeout, producing durations that are wrong in a way that is hard to notice: the operation appears to have taken as long as the export interval rather than as long as it ran.
End every span with a construct that guarantees completion on both paths - try-with-resources, using, defer, a context manager. Then record the outcome: set the span's error status and record the exception in the failure path. A span that ends with a default status after a failed operation is worse than no span, because it reports success.
This is the tracing-shaped version of the argument in error handling and exception discipline. The failure path is the path that matters, and it is the one least likely to have been exercised before it runs in production at three in the morning.
7. Trace Quality Is a Fleet Property
One service that drops context, floods cardinality, or ships with sampling at zero degrades every trace that passes through it, and the team that made the choice is not the team that pays. This is the incentive problem underneath every stalled tracing rollout. Instrumenting your service costs your sprint and benefits whoever debugs a cross-service latency question next quarter. Naming a span after the concrete request rather than the route template costs you nothing and makes the backend's aggregate views useless for everyone.
Name spans after the operation or route template, never the concrete request - an identifier, path parameter, or timestamp in a span name produces unbounded cardinality that degrades query performance for the whole backend. Never set a span attribute to an email address, name, national identifier, token, credential, or full request body; span attributes flow to a backend with a different access model than your database, which makes this the same exposure covered in PII leak prevention in logs, errors, and telemetry. Read the sampling rate and exporter endpoint from configuration resolved at startup, and treat a deployed sampling rate of zero as a defect.
Diagnostic worth running: pick your three highest-cardinality span names in the backend. If any of them contains an identifier, the team that produced it has no idea, because nothing in their own service degraded.
How Pandorian Enforces This Guide
Every principle above resolves to a shape in code: an entry point handler with no extract call, an outbound client constructed without a propagator, a publish without an injected carrier, a span started outside a try-with-resources, a span name built by concatenating an ID, a hardcoded sampling rate.
Caught at PR level
A new queue consumer that does not re-establish context, a span ended only on the return path, or a span attribute set to an email address surfaces with the line, the rule, and a suggested fix.
Full scans
An on-demand scan across the whole codebase answers the coverage question directly: which entry points establish context, which outbound calls propagate it, which asynchronous boundaries drop it. That inventory is the honest measure of whether tracing works, and no single team can produce it.
Scheduled scans
The same scan on a cadence, catching drift introduced outside any PR: a client library added to a shared factory without a propagator, a sampling rate changed during an incident and left at zero, a span name that grew an identifier through a refactor.
A Taste of the Guideline Package
Four of the checks that ship in this package - each runs at the surface where it catches the problem earliest.
FAQ
What is distributed tracing and correlation ID propagation?
+Distributed tracing records one request as it moves across services, linking each unit of work to the one that caused it. Correlation ID propagation is the mechanism underneath: an identifier generated at the edge and carried through every downstream call, queue message and job, so records emitted in different services can be stitched back into one story.
The propagation half is where implementations fail. Instrumenting a service to emit spans is straightforward. Ensuring the context survives every hop, including the ones that leave the HTTP path, is the actual work.
What is W3C Trace Context?
+W3C Trace Context is the standard that defines how trace identity travels between services, primarily through the traceparent header carrying a version, trace ID, span ID and sampling flag, plus an optional tracestate for vendor data. It replaced a landscape of incompatible formats (B3, X-Ray, Jaeger) that made traces break at the boundary between systems using different conventions.
It matters most when you integrate. A vendor, a service mesh and your own instrumentation only produce one continuous trace if all three speak the same propagation format.
What is the difference between a trace, a span and a correlation ID?
+A trace is the whole request journey, a span is one timed unit of work inside it with a parent pointer, and a correlation ID is a flat identifier shared by everything related to that request. Traces and spans give you causality and duration. A correlation ID gives you a filter.
They serve different tools. Spans populate your tracing backend and answer where time went. The correlation ID is what you put in every log line so that filtering logs by one value returns everything, including from services that were never instrumented for tracing.
Do you need distributed tracing if you already have good structured logging?
+For a small number of services, often not. Structured logs carrying a shared correlation ID answer most incident questions, cost far less, and require no additional infrastructure. Adding a tracing backend before you need it buys operational overhead in exchange for dashboards nobody opens.
The point where that stops holding is specific: when the question changes from what happened to where the time went. Logs tell you a request touched seven services and took four seconds. They will not tell you which of the seven consumed three of those seconds, or that two calls ran sequentially that could have run concurrently. Once request paths fan out and latency rather than errors is the recurring problem, correlated logs stop being sufficient.
How do I tell if our traces actually stitch together end to end?
+Pick your most complex user-facing request and open one real trace. Count the services it touches in the trace against the services you know it touches. The gap is your answer, and there is usually a gap.
Then check the four places context reliably drops: asynchronous boundaries where a message goes onto a queue and the consumer starts a fresh trace, background jobs and scheduled tasks that have no inbound request to inherit from, any service using a different propagation format than the rest, and calls made through an HTTP client that was constructed manually rather than through the instrumented one.
Can't OpenTelemetry auto-instrumentation handle propagation?
+Auto-instrumentation covers the common inbound and outbound paths of frameworks it recognizes, which is genuinely most of them, and then stops. It does not carry context into a thread pool you manage, into a message you publish to a queue with a custom serializer, into a job your scheduler starts, or through a client library it has no hook for. Every one of those is a place a trace ends silently, and nothing reports the break.
What is needed is a rule stating that context propagation is mandatory across every boundary a request crosses, including asynchronous ones, applied when the code is written rather than discovered during an incident. That is a codebase property: checkable on each pull request that adds a client, a producer or a job, and scannable across repositories for the boundaries that already drop it.
Get the Guideline Package
You'll get this as a guideline MD file - hand it straight to your own coding agent or CI setup. Or skip the DIY step: give these same guidelines real enforcement superpowers, with PR checks and scans live on your repos, through Pandorian.
From Principles to Enforcement
Every rule in this guide is easy to agree with in a design review and easy to violate six sprints later, one locally-correct decision at a time. Pandorian turns boundary discipline like this into CI guardrails - scoped to the repos, teams or services you choose.
