Secrets Management Best Practices: 7 Rules for the Full Lifecycle
Most organizations adopt a vault, declare the problem solved, then discover during an incident that the credential they need to revoke exists in nine other places - a CI variable, a state file, an image layer, a Slack thread from 2023.
TL;DR
- A secret that reached Git is compromised in every clone that ever existed, and the only remediation that means anything is rotation.
- A secret that enters a build artifact ships everywhere the artifact ships, which is the one place a vault cannot reach it.
- Every additional read of a secret is a new rotation obligation, so the number of call sites is the real measure of your exposure.
- A missing secret must stop the boot, because a placeholder default converts a configuration error into a silent security downgrade.
- Vaults protect secrets at rest while logs, process tables, and exception messages give them away at runtime.
- Infrastructure code that does not declare its own sensitivity leaks credentials into plan output, state files, and CI logs by default.
- Rotation spans every team that consumes a credential and belongs to none of them, which is why the secrets that cannot be rotated never are.
What Secrets Lifecycle Management Actually Covers
A secret is any value whose disclosure grants access: a password, API key, access token, private key, certificate key material, connection string carrying a credential, or webhook signing secret. Lifecycle management covers how those values are created, delivered, read, rotated, and revoked - not just where they sit when idle.
The distinction matters because most secret-handling programmes optimize the idle state. A vault is deployed, storage is encrypted, access is audited, and the security review passes. Then a credential leaks, and the expensive part of the incident is not the leak. It is the four days spent enumerating consumers, because rotating the value breaks an unknown number of services that copied it somewhere the vault does not know about. None of the seven principles below are about developers who do not care. They are about the fact that a secret's blast radius grows with every copy, and no single team can see the full consumer list.
1. A Secret in Git Is Already Compromised
Deleting a credential from HEAD does not remove it from history, and every clone that ever ran contains that history. This is the failure people most often mishandle, because the fix that feels correct - a commit removing the line - changes nothing about the exposure. The value remains in the object database, in every fork, in every CI cache, in every laptop that pulled that week.
STRIPE_KEY = "sk_live_51H8xQ2Kj9mNpQr..." # in a test fixture
STRIPE_KEY = "sk_test_FAKE_DO_NOT_USE"
Never write a secret as a literal in source, tests, infrastructure-as-code, config files, manifests, or documentation, and never commit a .env file with real values, a cloud credentials file, a service account key, a kubeconfig, or a private key. When one lands anyway, the response has exactly two steps: rotate the value, then remove it - in that order. The OWASP Secrets Management Cheat Sheet treats repository presence as disclosure regardless of visibility. And never resolve a scanner finding by suppressing it, which converts a detected exposure into an undetected one.
2. Secrets Resolve at Runtime, Not at Build
A secret that enters a build artifact travels everywhere that artifact travels, and no runtime control can call it back. The container image is the common case: a Docker ARG that carried a registry token persists in image metadata and layer history, readable by anyone who can pull the image. The same shape appears in a compiled binary with an embedded key and a Helm values file with a literal password.
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc \
&& npm ciRUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci
Obtain secrets at startup or first use by calling a secrets manager or parameter store client, or by reading a file mounted from one. Never declare a secret as a literal environment variable in a Kubernetes manifest, Helm values file, task definition, serverless config, or compose file - reference a store entry or a mounted secret file instead. Docker's own build secrets documentation is explicit that build arguments and environment variables are the wrong mechanism, because they persist in the final image.
3. Load Once, Reference Everywhere
Every additional place that reads a secret is a new rotation obligation, and the count of those places is the number nobody in your organization can produce. The pattern that creates them is innocuous: the connection factory reads DB_PASSWORD from the environment, then so does a migration script, then a health check, then a background worker. Each read is correct. Together they are why the rotation runbook has a step that says "grep."
# config.py — the only place this store path is read
DB_PASSWORD = secrets.get("prod/db/app-user")
# everywhere else
from config import DB_PASSWORDResolve each secret through a single named accessor or configuration property populated at startup, and reference that accessor wherever the value is needed. Reference the store entry by name or path so a rotated value is picked up on the next resolution - never copy the resolved value into a second store, a configuration file, or a deployment artifact.
Diagnostic worth running: pick your production database password and count the distinct places it is read from the environment or the store. If the answer is more than one, you have already decided that rotating it will be a project rather than an operation.
4. A Missing Secret Must Stop the Boot
A placeholder default converts a configuration error into a silent security downgrade, and the service will run happily in that state until something reads the data it was supposed to protect. The classic version is a signing secret that falls back to an empty string in a non-production branch and then ships: signature verification still runs, still passes, and verifies nothing.
key, err := secrets.Get(ctx, "payments/webhook-signing-key")
if err != nil {
log.Fatalf("required secret unresolved: %v", err)
}Fail application startup with an explicit error when a required secret cannot be resolved. Never supply an empty string, a placeholder, or a development default as a fallback. A service that refuses to start is a five-minute incident with an obvious cause. A service that starts with a blank signing key is a finding in someone's penetration test eighteen months later.
The twelve-factor guidance on config makes the underlying case: the litmus test is whether the codebase could be open-sourced at any moment without compromising a credential. A development default is a credential-shaped hole in that test.
5. Logs and Process Lists Leak What Vaults Protect
A secret resolved correctly from a vault and then printed into a log line has the same disclosure profile as one committed to Git, with worse discoverability. Three mechanisms produce almost all of it: a whole configuration object passed into a logging call, an authorization or cookie header captured by request logging, and an exception message that interpolates the value that failed to parse.
logger.info('service starting', { config });
// config carries dbPassword, signingKey, and every future fieldlogger.info('service starting', {
resolvedSecrets: ['prod/db/app-user', 'payments/webhook-key'],
});Never pass a secret, credential object, authorization header, cookie header, token, or whole configuration object into a logging call, exception message, string interpolation, or serialization function. And never pass a secret as a command-line argument to a process, container command, build step, or CI job: anything on the command line is visible in the host process table to every other process on the machine, and is routinely captured verbatim in CI job logs. This is the credential-shaped instance of PII leak prevention in logs, errors, and telemetry.
6. Infrastructure Code Declares Its Own Sensitivity
Infrastructure-as-code leaks credentials by default, because plan output, state files, and CI logs all render variable values unless something tells them not to. Terraform writes sensitive data as plain text into state, which AWS's own prescriptive guidance names directly as the reason state files require the same protection as the secrets inside them.
Mark every variable and output that carries a secret as sensitive - the sensitive attribute in Terraform, NoEcho in CloudFormation, the equivalent elsewhere. Then declare an explicit customer-managed encryption key reference on every secret, parameter, and Kubernetes Secret encryption configuration rather than relying on a platform default. Kubernetes stores Secret data unencrypted in etcd until encryption at rest is explicitly configured, which surprises a reliable percentage of teams during their first audit.
Marking outputs sensitive makes debugging a failed apply meaningfully harder, and someone will eventually propose unmarking one temporarily. That proposal is how the value ends up in a CI log with a ninety-day retention policy.
7. Rotation Is the Capability Nobody Owns
Rotation spans every team that consumes a credential and belongs to none of them, which is exactly why the secrets that cannot be rotated are the ones that never are. This is the principle that makes the previous six insufficient alone. Each team stored its secret correctly, resolved it at runtime, kept it out of logs. Then one shared credential - the broker password, the internal signing key, the partner API token - accumulated consumers across six services and two contractors, and rotating it now requires a change window nobody wants to schedule.
The consumer list is the artifact that does not exist. It cannot be produced from any single repository, because the shape you need is "which services resolve this store path," and that shape only appears from a view of the whole codebase. Reference secrets by store path so rotation is a store-side operation. Keep one accessor per secret per service so the consumer list is countable. Treat every scanner suppression as a deferred rotation with no owner and no date.
Diagnostic worth running: name the last secret your organization rotated on schedule rather than in response to an incident. If nobody can, rotation is not a policy you have - it is one you have written down.
How Pandorian Enforces This Guide
Every principle above resolves to a shape a scanner can see: a high-entropy literal in a manifest, a Dockerfile ARG consumed by a credential-shaped variable, a config object passed into a logging call, a Terraform variable with a secret-shaped name and no sensitive attribute, the same store path resolved at four call sites.
Caught at PR level
A new literal credential, a .env file added to the tree, or a secret passed as a CLI argument in a CI step surfaces with the line, the rule, and a suggested fix - before the value reaches a branch anyone else has pulled.
Full scans
An on-demand scan across the whole codebase sets the day-one baseline, including credentials that predate the guideline. It is also the only surface that produces the consumer list: every service resolving a given store path, in one view, which is the artifact rotation actually requires.
Scheduled scans
The same scan on a cadence, catching drift introduced outside any PR: a scanner suppression added during a release push, a secret copied into a second parameter store during a migration, an infrastructure variable that lost its sensitive marking in a refactor.
A Taste of the Guideline Package
Four of the checks that ship in this package - each runs at the surface where it catches the problem earliest.
FAQ
What is secrets lifecycle management?
+Secrets lifecycle management is the practice of controlling a credential across its whole life: how it is created, where it is stored, how an application receives it at runtime, how often it is rotated, and how it is revoked when someone leaves or a leak is suspected. Most organizations solve storage and stop there, which leaves rotation and revocation as manual processes nobody has run recently.
The test of a lifecycle is not where the secret sits. It is how long it takes to rotate every copy of one credential, and whether anyone knows how many copies exist.
What is secret rotation, and how often should it happen?
+Rotation is replacing a credential with a new value and retiring the old one, ideally with an overlap window so nothing breaks mid-cutover. Frequency depends on blast radius and on how the secret is delivered: short-lived dynamically issued credentials measured in hours, long-lived static ones typically 90 days or less.
The more useful question is not how often but how long it takes. If rotating a database password requires coordinating a deploy across six services, the interval on paper is irrelevant, because the real answer during an incident will be "we cannot do that today."
What is the difference between a secrets manager and environment variables?
+A secrets manager (Vault, AWS Secrets Manager, GCP Secret Manager) is a system of record that stores credentials encrypted, controls access per identity, logs every read and supports rotation and revocation. Environment variables are a delivery mechanism: a way of getting a value into a process at runtime.
They are not alternatives, which is where the confusion starts. A well configured system usually uses both, with the manager as the source of truth and the variable as the last hop. The problem is a manager that is bypassed, where the value originated in a CI setting or a .env file and no system of record knows it exists.
Are environment variables a safe place to store secrets?
+Safe enough as a delivery mechanism, not as storage. The environment is readable by the whole process, which means any dependency in your tree can read it, it is frequently captured by crash reporters and debug endpoints that dump the environment, and on some platforms it is visible to other processes on the host. Those are real risks, and they are also bounded ones.
The larger problem is upstream. When the environment variable is the source of truth, the secret's real home is a CI configuration screen, a Terraform variable file, or a .env on four laptops, with no access log and no rotation path. Injecting a short-lived value from a secrets manager into the environment at startup is a reasonable pattern. Typing a permanent value into a settings page is not, and the two look identical from inside the application.
How do I find secrets already committed to our repositories?
+Scan history rather than the working tree, because the interesting ones were removed in a later commit and are still recoverable from the object store. Tools like gitleaks or trufflehog can walk every commit on every branch. Include forks and archived repositories, since those are frequently outside whatever scanning was configured later.
Then widen past git. Check CI job logs for values echoed during a build, container images for build-time arguments baked into layers, infrastructure state files, and internal wikis. And treat every finding as live until proven otherwise: a leaked credential is compromised from the moment it was pushed, so remediation is rotation, not deletion of the commit.
Can't a secret scanner in CI solve this?
+A scanner catches high-entropy strings and known credential formats after the commit exists, which covers the obvious case and misses several that matter. It does not know that a config file references a value nobody rotated in two years, that a service reads a credential from an environment variable no secrets manager issued, or that a new deployment manifest introduced a plaintext value in a format the scanner has no pattern for.
Lifecycle is a property of how the code obtains and refreshes credentials, not of whether a string looks random. That needs the rule your organization actually wrote, checked on every pull request touching config or deployment, with scans establishing where the current bypasses live.
Get the Guideline Package
You'll get this as a guideline MD file - hand it straight to your own coding agent or CI setup. Or skip the DIY step: give these same guidelines real enforcement superpowers, with PR checks and scans live on your repos, through Pandorian.
From Principles to Enforcement
Every rule in this guide is easy to agree with in a design review and easy to violate six sprints later, one locally-correct decision at a time. Pandorian turns boundary discipline like this into CI guardrails - scoped to the repos, teams or services you choose.
