Secrets & KMS

Encryption-at-rest is mostly a checkbox unless you actually rotate keys. Rotation is the operational practice that turns a compliance bullet point into real defence in depth. The mechanics — KMS keys, data keys, envelope encryption, secret rotation — are the same shape on every cloud.

Analogy

Think of a hotel safe. Each guest gets a small safe in their room (the data key) for their valuables. The hotel master key (the KMS key) opens any guest safe but never leaves the manager's office. When a guest checks out, the room safe gets re-keyed (DEK rotation) so the next guest's belongings aren't accessible by the previous one's combination. The master key itself is rotated annually as a defence-in-depth move — even if a former employee remembers the old one, it's already useless. Encryption-at-rest without rotation is the hotel chain that brags every door has a lock, then never re-keys when staff leave.

Two AWS services for "I need to store secrets"

AWS Secrets Manager

For things that must rotate. Built-in rotation Lambdas for:

• RDS / Aurora primary credentials.
• DocumentDB / Redshift / DynamoDB.
• Custom secrets via your own rotation Lambda.

Features:

• Per-secret IAM policy.
• Automatic multi-region replication.
• Versioning (AWSCURRENT, AWSPREVIOUS, AWSPENDING stages).
• Built-in audit via CloudTrail.

Cost: ~$0.40/secret/month + per-API-call charges.

AWS Systems Manager Parameter Store

For config — sensitive or not — without auto-rotation. Two tiers:

• Standard parameters: free, up to 4 KB plaintext, 10,000 per region.
• Advanced: $0.05/parameter/month, up to 8 KB, supports policies (e.g. expiration).

Three types:

• String — plain text.
• StringList — comma-separated.
• SecureString — encrypted under a KMS key.

When to pick which:

Need	Service
Auto-rotated DB password	Secrets Manager
Static API token (no rotation)	Parameter Store SecureString
Hierarchical config (/app/prod/...)	Parameter Store
Cross-account / cross-region secret	Secrets Manager
Dirt-cheap non-secret config	Parameter Store standard (free)

The split is mainly cost vs rotation. If you don't need rotation, Parameter Store is functionally fine and dramatically cheaper.

AWS KMS — the core service

KMS keys (also called CMKs / Customer Master Keys) are stored inside AWS HSMs. The key material never leaves the HSM. Every cryptographic operation happens inside the HSM and returns the result.

The four key types:

• Customer-managed keys (CMK) — you own them, you set policies, you delete (with a 7–30 day waiting period).
• AWS-managed keys — created automatically by AWS services (aws/s3, aws/rds); you can use them, can't change their policy.
• AWS-owned keys — invisible; AWS uses them internally.
• Imported / external — you bring your own key material via BYOK.

For most apps you create one or a few customer-managed CMKs and grant per-resource access via IAM and key policies.

Envelope encryption — the fundamental pattern

You never encrypt large data with a KMS key directly. Two reasons:

1. KMS Encrypt is limited to 4 KB.
2. KMS calls cost money and rate-limit; encrypting 1 GB in 4 KB chunks would be ruinous.

Instead, envelope encryption:

1. Your app asks KMS: "give me a 256-bit AES data key"
   KMS returns: { Plaintext, CiphertextBlob }   ← same key, two forms
2. Your app uses Plaintext to encrypt the blob locally with AES-GCM
3. Your app stores: { encrypted_blob, CiphertextBlob_of_data_key }
   Your app forgets the Plaintext data key.

When you decrypt:

1. Your app reads { encrypted_blob, CiphertextBlob_of_data_key }
2. Your app asks KMS: "decrypt this CiphertextBlob"
   KMS returns the Plaintext data key (after checking IAM)
3. Your app uses Plaintext data key to decrypt the blob locally

KMS sees only the small data key, never the actual data. Most AWS services (S3 SSE-KMS, EBS, RDS, EFS) implement envelope encryption automatically. You provide the KMS key; the service handles the dance.

For client-side encryption, the AWS Encryption SDK does it for you: input plaintext + KMS key ARN; output a self-describing ciphertext that includes the wrapped data key.

Key rotation

KMS supports automatic annual rotation for symmetric customer-managed keys:

resource "aws_kms_key" "app" {
  description             = "App-data encryption"
  enable_key_rotation     = true
  deletion_window_in_days = 7
}

What rotation actually does: KMS generates new key material and stores it as a new version under the same key ID. Old data still decrypts because KMS keeps prior versions; new data is encrypted with the latest. From the application's perspective, nothing changes — same KMS key ID, just better hygiene.

Rotation is the difference between encryption-at-rest as a checkbox and encryption-at-rest as actual defence:

• Without rotation: a leaked key (insider, supply-chain, breach) is forever. Anyone with the key can read all historic data.
• With annual rotation: the leak window is bounded. After rotation, only future data is at risk.
• With per-tenant data keys + rotation: a leak of one tenant's key affects only that tenant.

Rotation is mandatory for SOC2, HIPAA, FedRAMP, PCI-DSS — but more importantly, it's what makes your security model actually work.

IAM scoping

The two layers of access control on a KMS key:

1. The key policy (JSON, attached to the key) — defines who can use the key. This is mandatory; an empty key policy locks even the account root out.
2. IAM policies on principals — refine within what the key policy allows.

The intersection rules: an action is allowed only if both the key policy and the principal's IAM policy permit it. Tightly-scoped key policies + generous IAM is a common pattern; the inverse (open key policy + tight IAM) is brittle.

Critical actions to scope:

• kms:Decrypt — read encrypted data.
• kms:Encrypt, kms:GenerateDataKey — write encrypted data.
• kms:DescribeKey, kms:ListAliases — read key metadata.
• kms:CreateGrant — delegate, time-limited.

The two patterns to internalise:

• Service principals for AWS services that need to use the key (s3.amazonaws.com, lambda.amazonaws.com).
• Encryption context — a key/value map you pass on every cryptographic call. KMS includes it as additional authenticated data; if it doesn't match on decrypt, the call fails. Use it to bind ciphertexts to a tenant ID, an object key, or a request scope. Free additional safety.

Common pitfalls

.env files committed to git. The original sin. Use Parameter Store / Secrets Manager + IAM-based fetch at boot.

Storing the data key alongside the wrapping key in the SAME bucket. A breach of the bucket gives the attacker both. Either separate buckets/accounts, or rely on KMS being external (which it is) plus IAM scoping.

Hardcoding KMS key IDs as env vars. Fine in dev; brittle in prod. Use Parameter Store or per-environment IaC outputs.

Caching decrypted secrets forever. A rotated secret won't take effect until your process restarts. For long-lived processes, either cache with a TTL (5–15 min) or use the rotation hooks AWS provides.

Skipping encryption context. Without it, a stolen ciphertext is decryptable anywhere with kms:Decrypt access. With encryption context bound to (tenant, resource), even a leaked ciphertext is useless out of context.

Using the default aws/secretsmanager AWS-managed key. It works, but you can't grant cross-account access to AWS-managed keys. Use a customer-managed CMK from day one if you'll ever need cross-account.

A reasonable production setup

• Secrets that rotate (DB passwords, OAuth client secrets) → Secrets Manager with rotation Lambdas.
• Static config + non-rotating tokens → Parameter Store SecureString under a customer-managed CMK.
• Encryption-at-rest for S3, EBS, RDS, EFS → SSE-KMS with a per-environment CMK + annual rotation enabled.
• Client-side encryption of PII or sensitive blobs → AWS Encryption SDK + per-tenant encryption context.
• Audit → CloudTrail logs every KMS call (decrypt, generate, create-grant) — review them.

That layout passes most compliance audits and gives real defence-in-depth, not just a marketing-page checkbox.