Secrets in Config

Every running service needs credentials — database passwords, API keys, signing secrets, OAuth client-secrets. Where you put those secrets is the difference between a routine release and a 3am page after some intern's pull request leaks your AWS root keys to a public fork. The patterns below are not negotiable.

The hierarchy

From worst to best:

Pattern	Verdict	Why
const DB_PASS = "..." in source	Never.	One forced push to a public fork → game over.
.env file committed to repo	Never.	Same as above with extra steps.
.env file gitignored, on dev machines	Local dev only.	OK for localhost:5432, never for prod.
Env vars baked into image at build	No.	Layer leaks them; everyone with image access has them.
Env vars from CI/CD secrets	OK for boot-time.	But you've now coupled secrets to your CI.
Secret manager + env vars at runtime	Yes.	Fetched at container start; nothing on disk.
Secret manager + dynamic / leased secrets	Best.	Vault-style: each instance gets a fresh, short-lived credential.

The destination is "secret manager + env vars at runtime." Get there in steps if you have to, but get there.

The 12-factor compromise

The original twelve-factor doctrine (Heroku, ~2012) said: store config in environment variables. That advice was right for its time and still useful today — but it doesn't tell you where the env vars come from. The implicit answer was "whoever launches the container sets them." For modern infrastructure, that means the secret manager populates them at container start.

The reason env vars won the API is that they're language-agnostic, deployment-mechanism-agnostic, and trivially overridable in tests:

# Production: secret manager → env var → your app reads process.env.DB_PASS
# CI:        GitHub secret  → env var → your app reads process.env.DB_PASS
# Local dev: .env file      → env var → your app reads process.env.DB_PASS
# Tests:    inline override → env var → your app reads process.env.DB_PASS

The app code is identical in every case. That's the point.

What the secret manager does

A real secret manager — Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Doppler — gives you four things you don't get from "files on disk":

1. Access control. IAM-style: this app role can read prod/db/password but not prod/payments/stripe-key. Audit log of every read.
2. Rotation. Built-in rotation hooks for common targets (RDS, RDS-Postgres, MongoDB Atlas). For custom systems, you write the rotator and call it from a Lambda / cron.
3. Versioning. Roll forward, roll back, "what was the secret 30 days ago" for incident-response.
4. Dynamic secrets (Vault's killer feature). Instead of a long-lived password, Vault generates a fresh credential per request with a short lease. Compromise becomes much less interesting because the credential expires before the attacker can do much.

The cost of running a secret manager is real — extra service, network call, configuration complexity. The cost of not running one is "every breach becomes an industry headline." Pick.

The .env-leak problem

Despite warnings, .env files end up in repos every day. Patterns that lead there:

• .gitignore exists but .env.production doesn't match the pattern.
• Developer copies .env into the repo "just for the commit" and forgets.
• A typo: .env-staging not in the ignore.
• Branch with .env survives and gets force-pushed to main during a merge.

GitHub has built-in secret scanning that flags well-known token patterns (Stripe, Slack, AWS) the moment they hit a public repo and notifies the issuing service to revoke them. Within 60 seconds of an AWS root key being committed, AWS itself has revoked it and emailed your account holder. This is wonderful — and it does not save you from the keys you'd never thought to register.

The fix is two-tier:

1. Pre-commit: gitleaks or trufflehog as a pre-commit hook. If the diff contains anything that looks like a secret, commit fails. Never reaches the developer's local repo, let alone GitHub.
2. Server-side / CI: Same scanners run in CI on every push. Commits with secrets are blocked from merging. Run on the full diff vs. main, not just the latest commit, so a secret hidden in a buried commit can't squeak through.

# Local pre-commit hook (.git/hooks/pre-commit)
#!/usr/bin/env bash
gitleaks protect --staged --redact -v

Plus a CI step that runs the same against PR diffs. Belt and suspenders.

The "encrypted in repo" trap

Tools like git-crypt, blackbox, and SOPS (with KMS) let you commit encrypted secrets to your repo. The encryption key lives elsewhere; the cipher-text is fine to commit.

This is fine for some scenarios (a small ops team, a single repo, a single environment) and a mess for others:

• Adding a new dev means re-keying the encrypted files for them.
• Removing a dev should mean rotating every secret they could read.
• The key itself has to live somewhere — a secret manager. So you've still got a secret manager, just hidden behind another layer.

For most modern teams, just use a secret manager directly. The "encrypted in repo" path was a 2015-era compromise born of "we don't have Vault yet."

What "right" looks like

For a typical web service:

1. Pre-commit secret scanner installed in every dev repo.
2. CI runs the same scanner on every PR diff vs. main.
3. Production secrets live in AWS Secrets Manager (or equivalent).
4. App role has read access to its own secrets only.
5. Container start reads secrets → exports as env vars → app reads process.env.
6. Local dev uses a .env file in .gitignore, with non-prod credentials.
7. Rotation cron rotates DB credentials on a schedule (90 days max).
8. Audit log of secret reads goes to your SIEM.

For a multi-service platform with stronger needs:

9. Vault-style dynamic secrets for the database — each app pod gets its own ephemeral DB credential.
10. mTLS between services with per-service cert issuance from a private CA / Vault PKI.
11. Break-glass procedure documented and tested: how to rotate every secret in 30 minutes.

What goes wrong (real examples, sanitized)

• Uber, 2016: AWS keys committed to a private GitHub repo. Attacker found credentials leaked to a different incident, got into the GitHub repo, found the AWS keys. 57M user records.
• Atlassian, 2021: A credential left in a CircleCI config from a long-running migration. Active for years.
• Toyota, 2022: Source code with a hard-coded T-Connect database access key, committed to a public GitHub mirror in 2017, discovered in 2022. ~300k customer records exposed.
• Microsoft, 2023: SAS token with overly broad permissions and a 2030 expiry committed to a public Azure DevOps repo. 38TB of internal data.

In every case the technical lesson is the same: secrets-in-repo is a time bomb whose fuse runs at the speed of your reorgs and forks. Don't ship the time bomb.

The conversation when you find one

You will, at some point, find a secret in a repo. Process:

1. Rotate immediately. Treat the secret as compromised regardless of repo visibility.
2. Log who could have read it. Audit the secret's usage from the moment it was committed.
3. Strip from history. git filter-repo or BFG to remove the secret from all commits, then force-push (with team coordination). This reduces the future exposure; the secret has already leaked, but at least the old history doesn't have it.
4. Post-mortem. Why did this happen, why didn't the scanner catch it, what change prevents the next one. Don't blame the developer who committed it; the system should have prevented it.

That's it. Rotate, audit, prune, learn. Don't pretend the secret is safe because the repo is private. Don't argue about whether it leaked. Treat it as compromised and move on.