Graceful Degradation When All Replicas Exceed the Lag Budget

Problem statement: Every replica has blown past its lag budget at once — a bulk import, a WAL sender stall, a network event — and your per-replica circuit breakers have all opened, leaving the read path with nowhere fresh to go without dumping the entire read load onto a primary that was never sized to absorb it.

Symptom Identification

This is the state the global breaker exists to handle, and it announces itself clearly:

  • Every per-replica breaker is OPEN at the same time, and the read_path_degraded signal from the circuit breaker patterns for degraded replicas guide flips to 1.
  • The moment the last replica is ejected, the primary’s connection count climbs steeply as reads that were being short-circuited all fall through to it.
  • FATAL: remaining connection slots are reserved for non-replication superuser connections starts appearing on the primary, and write latency rises because reads are now competing with the write path for connections.
  • Request latency spikes across the board, not just on read endpoints, because the shared primary pool is saturated — a read incident has become a write incident.

The failure to avoid is the naive one: “all replicas are stale, so send everything to the primary.” That converts a bounded staleness problem into an unbounded availability problem.

Root Cause Analysis

The primary cannot absorb the replica read load. A read fleet of three replicas may serve ten times the primary’s read capacity. When all three eject and every read falls through, the primary receives a multiple of the traffic it is provisioned for. Its connection pool exhausts, writes queue behind reads, and the database that was merely serving stale data now serves nothing. Fallback must be rationed, not unconditional.

Not all reads deserve the same fallback. Treating every read identically wastes the scarce primary capacity on cosmetic queries while starving the reads that actually need consistency. A balance check and a “related products” widget both became replica reads, but only one of them justifies spending a primary connection during a degradation event. Without a criticality classification, the router cannot prioritize.

There is no backpressure. If degraded reads queue against the primary instead of being shed, the queue grows without bound under sustained load, latency climbs until clients time out and retry, and the retries amplify the load — a classic congestion collapse. Graceful degradation requires the router to say no to some reads, quickly and cheaply. The complete degraded-state playbook this plugs into is covered in fallback strategies when replicas fall behind.

Architecture

Under degraded mode the router stops asking “which replica?” and starts asking “what is this read allowed to fall back to?” The answer is a function of the read’s criticality class.

Degraded-mode read routing by criticality When all replicas exceed the budget, an incoming read is classified. Critical reads go to the primary behind a rate limiter and are shed if the limiter is full. Degradable reads are served from cache with a staleness warning. Sheddable reads return a controlled empty or 503 response without touching the primary. Read all replicas open Classify criticality class critical degradable sheddable Primary rate-limited shed if full Stale cache + staleness warning Shed 503 / empty state

Step-by-Step Resolution

Step 1: Classify every read path by criticality

Criticality is an endpoint property, assigned once. Three classes are enough for most systems.

python
from enum import Enum

class Criticality(Enum):
    CRITICAL   = 1   # must be fresh or fail closed: balances, inventory, auth
    DEGRADABLE = 2   # may serve stale-with-warning: product info, counts
    SHEDDABLE  = 3   # may be dropped: recommendations, "people also viewed"

READ_CLASS = {
    "get_account_balance": Criticality.CRITICAL,
    "get_product_detail":  Criticality.DEGRADABLE,
    "get_recommendations": Criticality.SHEDDABLE,
}

Inline verification: every read endpoint resolves to a class — assert set(READ_CLASS.values()) <= set(Criticality) and that no critical endpoint is missing from the map in a startup check.

Step 2: Reserve and rate-limit primary read capacity

Decide how many of the primary’s connections may be spent on fallback reads, and enforce it with a token bucket so a burst cannot blow past the reservation.

python
import threading, time

class PrimaryReadBudget:
    """Token bucket: cap fallback reads/sec against the primary."""
    def __init__(self, rate_per_sec=200, burst=200):
        self.rate, self.burst = rate_per_sec, burst
        self.tokens, self.ts = burst, time.monotonic()
        self.lock = threading.Lock()

    def try_acquire(self):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.burst, self.tokens + (now - self.ts) * self.rate)
            self.ts = now
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

primary_budget = PrimaryReadBudget(rate_per_sec=200)

Size rate_per_sec from the primary’s spare read headroom, not its total capacity — leave room for writes. This is the same reservation discipline behind forcing primary reads for critical user transactions, applied fleet-wide during degradation.

Inline verification: hammer try_acquire() in a loop and confirm it returns True no faster than rate_per_sec on a sustained basis.

Step 3: Route by class in degraded mode

The degraded router spends the scarce primary budget only on critical reads, serves degradable reads from cache, and sheds the rest.

python
def degraded_read(name, query, cache, primary):
    klass = READ_CLASS[name]
    if klass is Criticality.CRITICAL:
        if primary_budget.try_acquire():
            return primary.execute(query)
        raise ServiceDegraded(retry_after=2)        # fail closed, not stale
    if klass is Criticality.DEGRADABLE:
        cached = cache.get(name, query)
        if cached is not None:
            return StaleResult(cached, stale=True)  # serve with warning flag
        # no cache: spend budget only if available, else degrade
        if primary_budget.try_acquire():
            return primary.execute(query)
        return StaleResult(None, stale=True)
    return SheddResponse(status=503, retry_after=5) # sheddable: drop cheaply

Inline verification: with the budget exhausted, a CRITICAL read raises ServiceDegraded, a DEGRADABLE read returns the cached value flagged stale, and a SHEDDABLE read returns 503 — none of them reaching the saturated primary.

Step 4: Surface staleness rather than hiding it

A degradable read served from cache must tell the caller it is stale so the UI can show an indicator and the client does not treat it as authoritative.

python
def render_product(result):
    body = serialize(result.value)
    headers = {}
    if result.stale:
        headers["X-Data-Freshness"] = "stale"
        headers["Warning"] = '110 - "Response is stale"'
    return Response(body, headers=headers)

Inline verification: during degradation, a get_product_detail response carries X-Data-Freshness: stale; when replicas recover it does not.

Step 5: Recover in priority order

As replicas catch up and their breakers close, restore traffic classes in order — critical to replicas first, sheddable last — keeping the primary rate limit in force throughout so a re-spike cannot stampede the primary.

python
RECOVERY_ORDER = [Criticality.CRITICAL, Criticality.DEGRADABLE, Criticality.SHEDDABLE]
# Re-enable each class only after replicas hold under budget for a sustained window,
# behind the slowstart ramp described in the replica-ejection guide.

Inline verification: confirm read_path_degraded returns to 0 only after sheddable reads resume, and the primary connection count never spikes during the handoff.

Configuration Snippet

Degraded-mode policy as configuration, tuned per environment:

yaml
degraded_mode:
  primary_read_budget:
    rate_per_sec: 200          # sized from primary spare read headroom
    burst: 200
  classes:
    critical:
      on_budget_exhausted: fail_closed      # never serve stale money-decisions
      retry_after_s: 2
    degradable:
      serve_stale_from_cache: true
      stale_header: "X-Data-Freshness: stale"
      cache_miss_fallback: primary_if_budget
    sheddable:
      action: shed
      status: 503
      retry_after_s: 5
  recovery:
    order: [critical, degradable, sheddable]
    require_under_budget_for_s: 30          # sustained catch-up before restoring

Verification and Rollback

Simulate a full-fleet stall and confirm the primary stays healthy:

bash
# Force every replica over budget at once
for r in replica_a replica_b replica_c; do
  psql -h $r -c "SELECT pg_wal_replay_pause();"
done

# Watch: primary connection count must stay bounded by the read budget
watch -n2 'psql -h primary -tc "SELECT count(*) FROM pg_stat_activity WHERE state=\"active\""'

The primary’s active connection count should plateau at roughly the write load plus rate_per_sec worth of critical reads — never climbing toward max_connections. If it climbs, the budget is not being enforced on some read path; audit for endpoints missing from READ_CLASS.

To roll back to non-degraded behavior after replicas recover, resume replay on every node and let the per-replica breakers close naturally; the router exits degraded mode when read_path_degraded returns to 0. If degraded mode misfires (e.g. shedding critical reads), the immediate mitigation is to raise rate_per_sec temporarily so more reads reach the primary, accepting higher primary load as the lesser evil while you diagnose.

Edge Cases and Gotchas

Cache stampede when the cache is also cold

Serving degradable reads from cache assumes the cache is warm. If a deploy or cache flush coincides with the replica stall, degradable reads miss, fall through to the primary-if-budget branch, and pile onto the already-scarce budget. Protect the cache with single-flight (request coalescing) so only one caller recomputes a given key, and consider serving a hardcoded default for sheddable-adjacent reads rather than any database hit.

Critical reads that are actually high-volume

A read tagged critical because it drives a money decision may also be high-volume (e.g. a checkout balance check on every page load). Under degradation its rate can exceed the primary budget and trigger fail_closed for real users. Split such endpoints: cache the read-mostly part, and reserve the primary budget only for the final pre-commit confirmation, which is where forcing a primary read for the critical transaction genuinely matters.

Silent degradation with no alert

If degraded mode is smooth enough that users barely notice, operators may not either — and the fleet can sit in primary-fallback for hours, one replica hiccup away from an outage. Alert on read_path_degraded == 1 for any sustained period as a page-worthy event, distinct from a single replica tripping, and track how long the system spends degraded as an SLO.

FAQ

Why not just send all reads to the primary when replicas are stale?

Because the primary is sized for write traffic plus a modest read margin, not for the full replica read load. Redirecting every read at once exhausts its connection pool, starves writes, and can take the whole database down — turning a stale-read incident into a full outage. Fallback to the primary must be rate-limited and reserved for reads that genuinely cannot tolerate staleness.

How do I decide which reads may serve stale data?

Classify read paths by consequence. A balance or inventory check that drives a money decision is critical and must reach fresh data or fail closed. A product description or a dashboard count is degradable and can serve a cached value with a staleness warning. Cosmetic or non-essential reads are sheddable and can return an empty or default state. The classification is a property of the endpoint, decided at design time.

How do I prevent a read stampede against the primary during recovery?

Recover traffic in priority order and cap the primary fallback rate the whole time. When replicas catch up, restore critical reads to replicas first, then degradable, then sheddable, each behind a slowstart ramp. Keep the primary fallback rate limit in force until replicas are comfortably under budget so a brief re-spike does not dump the full read load back onto the primary at once.

← Back to Circuit Breaker Patterns for Degraded Replicas