SLO-Based Alerting and Metrics Instrumentation: Why Most Alerts Watch the Wrong Thing

What Metrics, SLOs, and Alerting Instrumentation Actually Protects

Instrumentation discipline is a narrow slice of what code governance covers, but it's the slice that decides whether an incident is caught by an alert or by a customer. A gap in coverage rarely shows up as a clean failure. It shows up as an on-call engineer reconstructing an outage from logs because the one endpoint that broke never had a latency histogram, or a revenue-critical failure that never paged because every HTTP-level metric stayed green, the same boundary-instrumentation problem our distributed tracing guide covers for request flow instead of metrics.

None of the seven principles below are about developer carelessness. They're about the fact that a metrics and alerting system degrades by default the moment cardinality, ownership, or objectives aren't enforced as a contract. Unbounded label cardinality and alert thresholds with no declared objective are exactly the gaps discipline alone can't close.

1. Every Entry Point and Dependency Needs Instruments

A service ships a new endpoint, or a call to a new downstream, and nobody adds instrumentation because it "just wraps an existing one." Weeks later, that exact endpoint is the one degrading during an incident, with no data. Every entry point needs a request counter, error counter, and latency histogram; every outbound call needs a latency histogram and outcome counter.

REQUEST_LATENCY.labels(route="/orders").time()
REQUEST_ERRORS.labels(route="/orders").inc()

Instrumenting a shared client library once, instead of every call site,
means new dependencies get monitored automatically as they're added.

If a team can only measure a handful of things about a user-facing system, latency, traffic, errors, and saturation are the four worth measuring, applied at every boundary a request crosses.

2. Cardinality Kills Metrics Before Bugs Do

A team adds a user_id label to a histogram "for debugging." The metrics backend that tracked a few hundred time series now tracks millions, dashboard queries slow down organization-wide, and the monitoring system pages on-call for its own storage pressure. Never set a label to an identifier, a raw URL, or free text; use route templates and enumerated values.

requestDuration.WithLabelValues(r.URL.Path, userID)

requestDuration.WithLabelValues(routeTemplate, statusClass)

Every distinct label combination is its own time series with real memory and CPU cost. If a label's full set of values can't be enumerated before deploying, it doesn't belong in a label.

3. Metrics Declared Twice Are Already Wrong

Two files in the same service each define a counter for orders processed, with different names, because each was written at its call site without checking what already existed. Declare each metric once as a named constant, and register instruments once at module or startup scope, never inside a request handler or loop.

const REQUEST_DURATION = new Histogram({ name: "req_duration_seconds" });
// module scope, not inside the handler

A metric name should mean the same thing everywhere it's read: impossible if it's declared and named independently at more than one call site.

Constructed fresh inside a handler, a histogram silently resets its buckets and loses history on every call, a bug that never throws and just quietly produces useless data.

4. Green Dashboards Can Hide Broken Revenue

A payment webhook silently starts rejecting a subset of transactions after a schema change. Every HTTP-level metric looks perfect, because the endpoint still returns 200; the failure happens one step downstream, in business logic the request metrics never see. Emit a dedicated counter for every revenue-bearing operation, distinct from HTTP metrics.

An HTTP 200 means the server didn't crash. It says nothing about
whether the order that request represented actually completed.

The team finds out from a support ticket, not a dashboard, days after the revenue impact started.

A dedicated business-outcome counter costs one more metric per critical operation. That's a trivial cost against a silent revenue failure hiding behind a green dashboard.

5. An Alert Without an Objective Is a Guess

An alert fires on "error rate above five percent," a threshold picked at setup with no record of why. Six months later nobody remembers if it's strict or lax. Declare alert rules as code, and express every threshold against a named service level objective in the repository, never as a bare literal.

- alert: HighErrorRate
  expr: error_rate > 0.05

availability_slo: 0.999
expr: error_rate > (1 - availability_slo) * 14.4

An SLO measuring reliability the way customers actually experience it provides the highest-quality signal for deciding when an on-call engineer should respond, but only if the alert rule actually references it.

6. Alert on What Users Feel, Not Servers

An alert pages on a single instance's CPU crossing ninety percent while the load balancer already routed around it and no user noticed. A queue consumer falls behind for six hours with no lag alert, or a nightly job simply never runs. Alert on error rate, latency, and saturation of the user path, not a single instance's resources.

Every queue a service consumes needs a lag alert, and every scheduled job needs a completion heartbeat with an alert on its absence. A job that stops running entirely triggers no failure signal at all.

Node.js serves many clients with a small number of threads, so a callback that blocks one of them keeps that thread from handling any other client for as long as it stays blocked.

A short, severe burst of saturation can hurt users even while a longer-window utilization average looks completely fine.

7. Good Instrumentation Alone Doesn't Scale Across Teams

Team A instruments well by calling their monitoring vendor's client library directly. Team B, six months later, does the same with different naming and bucket conventions. Neither team did anything wrong, but the organization can't build a cross-service dashboard without a translation layer.

Emit metrics through the application's single shared instrumentation module, or a standard interface like OpenTelemetry, never by calling a vendor's client library directly from application code.

A shared naming scheme across a codebase and its libraries is what makes telemetry correlate and compose cleanly, and that only holds if every team routes through the same interface.

How Pandorian Enforces This Guide

Every principle above survives a design review. Few survive six months of independent team decisions, because instrumentation gaps aren't visible from inside any one pull request; they're visible only when an incident has no data behind it. Pandorian enforces this guide across three surfaces:

Caught at PR level

A new label bound to a raw identifier, a metric constructed inside a request handler, an alert threshold with no reference to a named SLO: each is a checkable code shape, flagged with the line, the rule, and a fix before it merges.

Full scans

New code discipline doesn't retroactively fix the endpoint from two years ago with no error counter, or the label quietly carrying a user ID. A full scan establishes the baseline across the whole codebase.

Scheduled scans

A new team's service brought online with its own vendor SDK calls, an alert threshold edited directly in a vendor console and never backported, a scheduled job with no heartbeat. Scheduled scans catch drift PR checks, by construction, never see.

A Taste of the Guideline Package

Four of the checks that ship in this package. Each runs at the PR level, on full or scheduled scans, or several at once.

FAQ