Structured Logging Best Practices: A Field Guide to Log Hygiene

What Structured Logging Standards Actually Protect

Ask five engineers why a credential ended up in a log store and four will describe a reasonable debugging shortcut, not carelessness. Logging the whole request object to see what a client sent, or catching an exception and logging just its message because the object felt redundant, is the fastest way to see what's wrong in the moment someone needs to see it.

None of the seven principles below are really about discipline. They're about the fact that logging correctly is easiest exactly when nothing is on fire, and hardest exactly when something is - which is the one moment none of these rules can be allowed to depend on memory.

logger.info(f"User {user_id} failed checkout: insufficient funds for order {order_id}")

logger.info("checkout.declined", user_id=user_id, order_id=order_id, reason="insufficient_funds")

The tradeoff: a stable message template means you can't improvise wording mid-incident to add color - you extend the field set instead. That's the correct trade. Expressiveness at the call site costs queryability at the aggregator, and the aggregator is what everyone actually uses six months later.

1. Log Records Are Data, Not Prose

Your on-call engineer is trying to find every instance of a failed checkout this week. The log line reads free text with the user ID and order ID woven in, so there's no field to filter on but a regex. Worse, the next engineer who touches this code rephrases the message slightly, and the same event now produces two different strings in the aggregator - the dashboard built to count this failure silently starts undercounting it the day the wording changed.

logger.info(f"User {user_id} failed checkout: insufficient funds for order {order_id}")

logger.info("checkout.declined", user_id=user_id, order_id=order_id, reason="insufficient_funds")

The tradeoff: a stable message template means you can't improvise wording mid-incident to add color - you extend the field set instead. That's the correct trade. Expressiveness at the call site costs queryability at the aggregator, and the aggregator is what everyone actually uses six months later.

2. Every Record Needs Its Own Coordinates

Three checkout requests hit your payment service in the same 200-millisecond window. All three fail. The logs interleave with no way to tell which line belongs to which request. Separately, a regression appears the morning after a deploy, but the log lines don't say which build produced them, so the first twenty minutes of the incident are spent confirming which of the last three releases is even running right now.

const logger = require('pino')();
app.use((req, res, next) => {
  logger.info(`handling ${req.method} ${req.path}`);
  next();
});

const baseLogger = require('pino')({
  base: { service: 'checkout-api', version: process.env.BUILD_SHA, env: process.env.NODE_ENV }
});
app.use((req, res, next) => {
  req.log = baseLogger.child({ correlationId: req.headers['x-request-id'] ?? crypto.randomUUID() });
  req.log.info('request.received', { method: req.method, path: req.path });
  next();
});

This is the same problem OpenTelemetry's logs specification solves at the platform level, by including trace and span identifiers directly in the log record. The tradeoff: correlation only survives if it's propagated across every hop, including the ones that don't look like hops — a queue message, a scheduled job. Miss one async boundary and the chain breaks silently.

3. Severity Levels Are Promises, Not Feelings

Every declined checkout - a customer with insufficient funds, not a bug, not anyone's fault - gets logged at error level, and error level pages someone. Within a month the on-call rotation learns to treat these alerts as noise, and three weeks later a genuine payment-processor outage gets the same shrug. Separately, a handler logs only the exception's string form, so during the incident this line was meant to help with, there's no stack trace and no cause chain - because the exception object itself was never passed to the logger.

try:
    charge_card(order)
except PaymentDeclined as e:
    logger.info("checkout.declined", order_id=order.id, reason=e.code)
except Exception:
    logger.exception("checkout.failed", order_id=order.id)

This is the exact discouragement test Google's SRE book describes for alerting design: a good alerting rule should ask whether the responder will ever be able to ignore it, knowing it's benign. An error level nobody can ignore, attached to a condition that happens constantly and requires nothing, fails that test immediately.

The tradeoff: down-leveling expected failures means you lose the ability to alert on their volume using log-based tooling alone. If you need to know when decline rate spikes, that's a metric with a counter, not a log level chosen to make a dashboard light up.

4. Log Volume Is a Budget You Set

A nightly batch job processes two million records and logs one line per record to track progress. On a normal night this is merely wasteful. On the night the job fails partway through - the exact night anyone needs the logs - the volume spike triggers the collector's rate limiter, and the records that would show exactly where processing stopped get dropped before they're indexed. Separately, a debug level added during a hotfix six months ago never got removed, and a production service has been logging at debug volume ever since.

logging.getLogger().setLevel(
  os.environ.get("LOG_LEVEL", "INFO")
)
# not: logging.getLogger().setLevel(logging.DEBUG)

Diagnostic worth running: pull up your log ingestion dashboard next to a batch job's record-count metric. If the two lines move in lockstep, every record is generating a log line, and the job you most need visibility into during a slow week is exactly the one whose logs your collector will start dropping first.

The tradeoff: a summary line per operation means losing per-record detail for the rare case you genuinely need it. Solve that with sampling — log one in a thousand records at full detail — rather than reverting to logging every record because sampling felt like more setup work.

5. The Logging Framework Is Your Only Exit

An engineer debugging a cache issue adds a quick print statement to see what's happening, ships it in the same PR, and it sits in production for eight months, breaking the log collector's JSON parser on every request. Separately, someone configures a rotating file handler to get logs on disk for one investigation. Nobody removes it. The container's local disk fills up over the following quarter, and the service starts failing because it can no longer write its own logs.

print(f"Cache miss for key {key}")
# ...
handler = logging.handlers.RotatingFileHandler(
  "/var/log/app/app.log", maxBytes=10_000_000
)
logger.addHandler(handler)

logger.debug("cache.miss", key=key)
# ...
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)

This is the same discipline behind the Twelve-Factor App's logs factor: a well-behaved process treats its logs as an unbuffered event stream written to stdout and never concerns itself with routing or storage. The only real pushback is wanting a readable console format in local development — which is a formatter setting on the same logger, not a reason to open a second, unmanaged exit path out of the process.

6. What You Log Becomes Your Attack Surface

An engineer debugging an authentication issue logs the entire incoming request object, headers included, to see exactly what the client sent. The bug gets fixed. The logging call doesn't. Six months later, a routine access review discovers that every bearer token issued during that window is sitting in plaintext in a log store with a 90-day retention policy and a longer list of people who can query it than the list who can touch the production database. No breach occurred. The credentials were compromised anyway, by the debugging convenience that logged them.

Diagnostic worth running: can someone with only log-viewer access reconstruct a user's password reset flow well enough to hijack the account? If the answer is yes, the problem isn't your access control policy - it's that the reset flow's logging is generating a credential-recovery script for anyone who can read logs.

The OWASP Logging Cheat Sheet frames this as sanitizing event data before it's written, not restricting who can read it afterward. MITRE tracks this exact failure mode as CWE-532, insertion of sensitive information into a log file — a named, catalogued weakness, not an edge case. It makes local debugging slightly less convenient: you decide beforehand which fields matter, instead of dumping the object and sorting it out later.

7. Redaction Can't Live at the Call Site

Six months after "never log a credential" becomes team policy, a scan turns up a dozen violations anyway - not from anyone ignoring the rule, but from a new hire who never read the wiki page, a hotfix shipped under incident pressure that skipped review, and a third-party SDK's debug-request-logging flag someone enabled during a support case and never disabled. Every violation is a reasonable individual decision made by someone who, if you asked them, would agree completely with the rule they just broke.

Rule of thumb: a logging standard that lives in a wiki page enforces nothing on its own; a redaction policy that lives in the logger's serializer enforces itself, on every call site, including the ones nobody's touched since before the policy existed.

NIST's guidance on log management makes the same structural point at the program level: effective log management requires consistent processes across an entire organization, not local decisions at each system. This is the honest hinge in this guide — every other principle can be applied correctly by a disciplined engineer at the moment they write the code. This one can't, because "at every call site, forever" isn't a property any individual's memory can hold at scale. That's what makes this a codebase governance problem instead of a code review problem, and it shows up fastest wherever AI-generated code is landing print statements and full-object logging back into a codebase faster than any human reviewer can catch them one PR at a time.

How Pandorian Enforces This Guide

Writing these seven principles down changes nothing on its own - every team already has some version of them somewhere. What changes outcomes is running them against real code, continuously, on Pandorian's three enforcement surfaces:

Caught at PR level

Every pull request is checked against the full rule set before merge. A string-interpolated log call, a print statement, a hardcoded debug level, a credential passed to a logger - each flagged with the line, the rule, and a fix.

Full scans

An on-demand sweep across the entire codebase establishes the day-one baseline: every existing file appender nobody's removed, every exception logged by message only, every call site where a request body gets logged whole - the violations no PR check could have caught because they predate the guideline.

Scheduled scans

The same full scan, run on a cadence, catches drift introduced outside any pull request - a dependency update that quietly re-enables verbose request logging by default, an infra config change that flips a log level, an SDK's debug flag left on after a support session.

A Taste of the Guideline Package

Four of the checks that ship in this package - each runs at the PR level, on full or scheduled scans, or several at once.

FAQ