CDNs and Anycast

A CDN's job is to put a copy of your content close to every user. Two technologies carry the load: anycast for getting the user to the nearest server, and caching hierarchies for getting the cached object to that server with as few origin fetches as possible.

Anycast — one IP, many locations

In normal (unicast) routing, an IP address belongs to one machine. In anycast, the SAME IP is announced from many physical locations via BGP. Routers pick whichever location is "nearest" by AS-path length and IGP cost.

                          POP1 (Frankfurt)
                            ▲   announces 192.0.2.1
User in Berlin ─[BGP picks closest]
                            ▼
                          POP2 (Tokyo)
                              announces 192.0.2.1
                          POP3 (São Paulo)
                              announces 192.0.2.1

The user's traffic goes to whichever POP is reachable by the fewest hops — usually but not always the geographically closest. Cloudflare announces 1.1.1.1 from 300+ cities; whichever you ping is "your" POP for that moment.

The brilliant part: failover is automatic. If a POP dies, BGP withdraws the announcement; routers re-converge in seconds; clients land on the next-nearest POP without changing IP.

POPs (Points of Presence)

A POP is a CDN's edge box. Physically: a rack in a colo (or a megacomplex of racks) connected to local internet exchanges. Logically: a caching reverse-proxy with a few thousand of its closest friends.

POPs do roughly four jobs:

1. Terminate TLS for clients (saves origin a CPU cost; lets the CDN see and modify HTTP).
2. Serve cached objects that match the request.
3. Forward cache misses upstream — to a shield, then to origin.
4. Run edge code (Workers, Functions, Lambda@Edge) that can transform requests/responses without round-tripping origin.

Cache hierarchy — POP, shield, origin

A naive setup has every POP talking directly to origin. With 300 POPs, a single origin fetch can be requested 300 times in parallel after a deploy invalidates the cache. The fix is a regional shield:

User ─▶ POP (any of 300)
            ▼ miss
        Shield POP (per-region, ~5 of them)
            ▼ miss
        Origin (single)

The shield's job is to absorb misses from the POPs so origin sees one fetch per (key, region) pair instead of dozens. Most modern CDNs (Fastly, CloudFront) let you turn this on per-domain.

Request collapsing

Even within a single POP, ten thousand users requesting the same uncached object simultaneously could trigger ten thousand origin fetches. CDNs collapse these:

t=0  user1 ──▶ POP miss ──▶ origin fetch starts
t=0  user2 ──▶ POP (in-flight detected, wait)
t=0  user3 ──▶ POP (in-flight detected, wait)
...
t=200ms     origin returns; POP serves all three from the buffer

This is "request collapsing" (Fastly's term) or "request coalescing" (everyone else). It bounds origin load to one fetch per cache key per POP, regardless of fan-in.

Pull vs push CDNs

Two ways to populate a CDN cache:

• Pull (the modern default): the CDN never has the object until a user asks for it. Miss → fetch from origin → store at the POP. The TTL governs how long it stays. Origin remains the source of truth.
• Push (older model): you upload every asset to the CDN explicitly — usually via FTP/S3/API. The CDN never talks to origin.

Pull won because it's strictly more flexible: short TTLs let you change content; long TTLs (with hashed URLs) get push-like performance for static assets. Push is now mostly a video/large-asset niche where you want guaranteed pre-warming.

Cache key and surrogate keys

Two requests are "the same object" only if they have the same cache key. Default cache key is (host, path, query), plus the Vary header (which says "this response varies by Accept-Language" or whatever).

A surprising amount of CDN tuning is about cache keys:

• Strip cookies from the cache key for static assets, or you'll have N copies for N users with different session cookies.
• Strip query parameters that don't affect the response (?utm_source=...).
• Add custom headers to the key when you serve different responses for different versions of the same URL (geo-personalisation, A/B tests).

For invalidation, surrogate keys (Fastly) or cache tags (Cloudflare) let you tag responses with arbitrary identifiers and purge by tag rather than URL:

Origin sets:  Surrogate-Key: product-42 category-shoes

Code change updates product 42:
  PURGE /service/X/purge/product-42  →  every cached response tagged
                                         product-42 is invalidated globally

Without surrogate keys, you're stuck with URL-by-URL purging — fine for a few invalidations, terrible for a sale that affects a million product pages.

What POP wins matter

CDN marketing emphasises POP count. Real performance depends on:

• Anycast quality — does BGP actually route you to the nearest POP, or to a regional one? The big four (Cloudflare, Fastly, Akamai, AWS) do this well.
• Peering — does the CDN peer directly with major eyeball ISPs (Comcast, Vodafone, etc), so traffic stays off transit?
• Hit rate — what percentage of requests are served from POP cache without going to origin? 95% is good; 99%+ is enviable.
• Origin-shield strategy — protects origin during deploys.
• Compute at the edge — Cloudflare Workers, Vercel Edge, Fastly Compute@Edge let you run code at every POP without origin round-trips.

Failure modes

Origin overload during invalidation. You purge a million keys; every POP gets misses; origin melts. Mitigation: surrogate keys instead of URL purges, plus a shield POP with rate-limiting upstream.

The "global outage from one bad config". A bad config deployed to all POPs simultaneously is a global outage. Mitigations: staged rollout (POP-by-POP), config validation in CI, kill-switches.

Anycast hop weirdness. Sometimes BGP doesn't route you to the closest POP. A user in Berlin might land in Frankfurt 99% of the time and Madrid the other 1%. Almost never an actual problem; debugging requires traceroute from the user's network.

TLS termination at the edge changes the trust boundary. The CDN sees plaintext after TLS terminates. For most apps that's fine; for HIPAA/PCI you might require the CDN's "encrypted between you and origin" tier.

Summary

CDNs distribute content via:

1. Anycast — one IP routed to the nearest POP via BGP.
2. POPs — edge boxes that terminate TLS, cache, and run edge code.
3. Shields — regional caches that absorb misses from POPs.
4. Request collapsing — one origin fetch per (cache-key, POP) regardless of fan-in.
5. Pull, almost always — origin is source of truth; TTLs govern freshness.
6. Surrogate keys for tag-based invalidation.

When you measure performance, measure hit rate first. Anything below 95% means you're paying CDN dollars for less benefit than you should be — fix the cache key.