Circuit Breaker Patterns for Degraded Replicas

A single slow replica should never be allowed to keep absorbing reads. When a node’s replication lag drifts past your freshness budget, or its connections start timing out, every request the router still sends there either returns stale data or stalls behind an apply queue. The circuit breaker is the control-theory answer to this: a small state machine that watches a replica’s health signals, trips the moment they cross a threshold, stops routing to the node while it recovers, and cautiously re-admits it only after a probe confirms it is healthy again. This guide covers how the breaker pattern maps onto read-replica routing, where it sits relative to your connection pools and proxies, and how to tune it so it protects your SLA without flapping.

Concept Definition and Scope

A circuit breaker for replica routing is a per-target health gate with three states — CLOSED, OPEN, and HALF-OPEN — that decides whether a given read may be dispatched to a given replica. In the CLOSED state, reads flow normally to the replica and the breaker counts failures. When failures cross a trip condition, the breaker moves to OPEN and short-circuits: reads are routed elsewhere (another replica, or the primary) without even attempting the degraded node. After a cooldown, the breaker enters HALF-OPEN and admits a single probe read; success returns it to CLOSED, failure sends it back to OPEN with a longer cooldown.

This borrows directly from the resilience pattern used for downstream service calls, but the failure signal is different. A service breaker usually trips on error rate and latency. A replica breaker trips on those plus a domain-specific signal that has no analogue in generic RPC: staleness. A replica can answer every query in two milliseconds with a 200 OK and still be returning data that is forty seconds behind the primary. That is a silent correctness failure, not a latency failure, so the breaker must treat a lag-threshold breach as a first-class trip condition alongside connection errors and health-check failures.

The scope of this pattern is the read path only. Writes always go to the primary and are outside the breaker’s concern. What the breaker governs is which read-serving nodes are eligible at any moment, and — through its global variant — what the read path degrades to when no replica is eligible at all. The three guides in this topic area go deeper on each layer: the application-code breaker implementation, infrastructure-level ejection and reintroduction via proxies, and graceful degradation when the whole fleet is stale.

Mechanism Deep-Dive

The heart of the pattern is the transition logic between the three states. The diagram traces every edge and the signal that triggers it.

Trip conditions

Three independent signals should be able to trip a replica breaker, combined with OR:

• Lag-threshold breach. The replica’s measured lag exceeds the tier’s freshness budget for N consecutive health checks. Consecutive, not instantaneous — a single scrape can catch a transient apply spike or a stale metric, and ejecting on one sample causes needless flapping. This signal is what makes a replica breaker different from a generic service breaker, and it depends on the lag telemetry described in detecting and handling replication lag in real time.
• Error rate. The fraction of reads to this replica that fail (connection refused, timeout, SSL connection closed) over a sliding window exceeds a ceiling, typically 20–50%. A rolling window matters here: a rate computed over a fixed reset interval hides a burst that straddles the boundary.
• Health-check failure. An out-of-band probe — SELECT pg_is_in_recovery() plus a lag query, or a proxy’s TCP/SQL check — fails or times out. This catches a node that has stopped answering entirely, which the error-rate signal would also catch but more slowly under low read volume.

Cooldown and backoff

When the breaker opens, it records the time and refuses the replica until a cooldown elapses. A fixed cooldown works, but a replica that opens, gets probed, and immediately fails again will thrash. Apply exponential backoff on repeated probe failures — cooldown = base * 2^(consecutive_probe_failures) capped at a ceiling — and add jitter so a fleet of application instances that all tripped on the same lag event do not synchronize their probes into a thundering herd against the recovering node.

Half-open probing

HALF-OPEN is the single most important state for avoiding a false recovery. The breaker admits exactly one read (or a small fixed number) to the previously-degraded replica and watches the outcome. Crucially, the probe must re-evaluate the same signal that tripped the breaker: if the trip reason was lag, the probe checks current lag, not just whether a connection succeeds. A replica whose TCP listener is up but whose apply queue is still forty seconds deep must fail the probe. Only when the probe confirms the node is within budget does the breaker close and resume full traffic.

Per-replica vs global breakers

These are two layers, not two alternatives. A per-replica breaker governs routing to one node and lets you eject a single degraded replica while healthy peers keep serving. A global breaker sits above the fleet and trips only when every per-replica breaker is open — that is the signal to enter a degraded read mode that falls back to the primary with load shedding, covered in the graceful-degradation guide. Running only a global breaker is a common mistake: one slow node then trips reads for the entire fleet.

Interaction with connection pools and load balancers

A breaker does not move traffic by itself — it changes a decision that some routing component then acts on, and where that component sits determines what “open” actually does. Three integration points matter.

Connection pools. When a per-replica breaker opens, the pool for that replica should stop handing out connections, but it must not slam existing checked-out connections shut mid-query — that turns a staleness problem into a burst of query errors. Drain, don’t kill: mark the pool ineligible for new checkouts, let in-flight queries finish, and recycle idle connections on their normal lifetime. The pool’s own validation query (a lightweight SELECT 1 on checkout) is a second, faster trip signal that complements the lag poll, and sizing the primary pool to absorb the redirected read load is a prerequisite covered in the connection pool architecture for read replicas.

Load balancers. A TCP or HTTP load balancer in front of the replicas already has a health-check and ejection mechanism — rise/fall counters, drain/down states, slowstart. The breaker’s job here is to feed the balancer a freshness-aware verdict rather than a bare liveness check, so the balancer ejects a lagging-but-reachable node. The balancer then provides the hysteresis and warm-up for free. This is why a proxy breaker and an application breaker are not redundant: the balancer enforces one ejection decision for every client, while the in-process breaker adds per-request consistency context the balancer cannot see.

Ordering. The breaker check must happen before pool checkout, not after. Acquiring a connection and then discovering the replica is tripped wastes a pool slot and adds latency; the eligibility decision belongs at the top of the routing path so a tripped replica is skipped without ever touching its pool.

Trade-Off Comparison Table

Where the breaker lives determines what signals it can see and how fast it reacts.

The application-level breaker reacts instantly and sees request context but only has one process’s view of the world; a proxy breaker has a fleet-wide view but reacts at the granularity of its health-check interval. High-scale deployments run both — a proxy breaker for coarse ejection and an application breaker for per-request consistency overrides — the same layering described in the broader connection routing and pooling strategy.

Configuration Runbook

Application-level breaker (pseudocode)

The logic below is framework-agnostic; the application-code guide turns it into a concrete, thread-safe implementation.

# Per-replica breaker evaluated at routing time
STATE = {"closed": 0, "open": 1, "half_open": 2}

def route_read(replica, now):
    b = replica.breaker
    if b.state == "open":
        if now - b.opened_at >= b.cooldown:
            b.state = "half_open"          # admit exactly one probe
        else:
            raise ReplicaUnavailable       # short-circuit, router tries next
    if b.state == "half_open" and not b.probe_slot.acquire():
        raise ReplicaUnavailable           # another request holds the probe

    lag_ms = replica.measured_lag_ms()
    if lag_ms > b.trip_lag_ms:
        b.record_breach()                  # consecutive-breach counter
        if b.breaches >= b.trip_threshold or b.state == "half_open":
            b.open(now)                     # backoff applied inside open()
        raise ReplicaUnavailable
    # healthy sample
    if b.state == "half_open" and lag_ms <= b.reenter_lag_ms:
        b.close()                          # hysteresis: lower re-entry threshold
    b.record_success()
    return replica.connection()

Key parameters and safe defaults for a user-facing tier:

replica_breaker:
  trip_lag_ms: 500          # tier freshness SLO
  reenter_lag_ms: 200       # hysteresis: must be well under trip to close
  trip_threshold: 3         # consecutive breaches before OPEN
  error_rate_threshold: 0.25
  error_window_s: 30        # rolling, not fixed-reset
  cooldown_base_ms: 2000
  cooldown_max_ms: 60000    # exponential backoff ceiling
  cooldown_jitter_ms: 500
  half_open_probes: 1

Proxy health-check approach (HAProxy + agent-check)

A proxy breaker is expressed as a health check whose result ejects the backend. HAProxy’s agent-check calls a lightweight agent on the replica that returns up, drain, or down based on measured lag, which HAProxy then acts on without ejecting for a single blip because of the rise/fall counters:

backend pg_read
    balance leastconn
    option httpchk GET /health
    # rise/fall provide the consecutive-sample hysteresis
    default-server inter 2s fall 3 rise 5 slowstart 30s
    server replica_a 10.0.1.11:5432 check agent-check agent-port 9101
    server replica_b 10.0.1.12:5432 check agent-check agent-port 9101

The agent returns drain when lag crosses the budget (stop new sessions, keep existing) and down when the node is unreachable; rise 5 means five consecutive healthy checks are required to close, fall 3 means three consecutive failures to eject, and slowstart 30s ramps weight back up over thirty seconds after recovery — the exact ejection-and-reintroduction mechanics covered in the replica-ejection guide. The fall/rise asymmetry is the proxy’s built-in hysteresis, and slowstart is its half-open probe.

Monitoring and Alerting Signals

Every breaker transition must be observable, or a fleet silently running in degraded mode looks identical to a healthy one. Expose at minimum:

• replica_breaker_state{replica} — 0=CLOSED, 1=OPEN, 2=HALF_OPEN. The single most important panel.
• replica_breaker_trips_total{replica, reason="lag|error_rate|health_check"} — counts trips by cause so you can tell a lag problem from a connectivity problem.
• replica_breaker_probe_total{replica, result="success|fail"} — half-open outcomes; a high fail ratio means the cooldown is too short for the recovery time.
• replica_measured_lag_ms{replica} — the raw signal, so you can correlate a trip with the lag curve that caused it.
• read_path_degraded — the global breaker’s derived state; 1 when all replicas are open and the primary-fallback path is active.

groups:
  - name: replica_circuit_breaker
    rules:
      - alert: ReplicaBreakerOpen
        expr: replica_breaker_state == 1
        for: 60s
        labels: {severity: warning}
        annotations:
          summary: "Replica {{ $labels.replica }} breaker OPEN — ejected from read pool"
      - alert: AllReplicaBreakersOpen
        expr: max(read_path_degraded) == 1
        for: 30s
        labels: {severity: critical}
        annotations:
          summary: "All replica breakers OPEN — reads falling back to primary"
      - alert: BreakerFlapping
        expr: rate(replica_breaker_trips_total[10m]) > 6
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "Replica {{ $labels.replica }} tripping >6x/10m — tune hysteresis/cooldown"

Alert on the derived global state separately from individual replica trips. A single replica opening is routine and self-healing; the whole fleet opening is a page-worthy incident because the primary is now absorbing read load it was never sized for.

Failure Modes and Recovery Steps

1. Breaker flapping between OPEN and CLOSED

Cause: The trip and re-entry thresholds are identical, so a replica hovering right at the budget oscillates every health-check interval. Recovery: Introduce hysteresis — set reenter_lag_ms well below trip_lag_ms (e.g. 200 vs 500), require multiple consecutive healthy probes before closing, and apply exponential backoff to the cooldown so repeated failures widen the retry gap. Confirm the fix by watching replica_breaker_trips_total flatten.

2. False recovery on a shallow probe

Cause: The half-open probe only checked that a TCP connection succeeded, so a replica with a live listener but a deep apply queue closed the breaker and immediately started serving stale reads. Recovery: Make the probe re-evaluate the trip signal — if the breaker tripped on lag, the probe must query current lag (pg_last_wal_replay_lsn position or heartbeat delta) and require it under reenter_lag_ms before closing.

3. Global breaker trips on one slow node

Cause: Only a single fleet-wide breaker was configured, so one degraded replica short-circuited reads for every node. Recovery: Split into per-replica breakers for routing plus a global breaker that trips only when all per-replica breakers are open. The per-replica layer ejects the one bad node; the global layer stays closed while healthy peers absorb the traffic.

4. Thundering herd against a recovering replica

Cause: Many application instances tripped on the same lag event and, with identical fixed cooldowns, all probed the recovering node at the same instant, re-saturating it. Recovery: Add jitter to the cooldown and cap half-open concurrency to one probe per instance. At the proxy layer, slowstart ramps weight gradually so the node is not hit with full traffic the moment it passes its first check.

5. Breaker masks a real capacity shortfall

Cause: Replicas repeatedly trip on lag because the fleet is genuinely undersized for write volume, and the breaker keeps shunting load to the primary, which hides the underlying problem until the primary saturates. Recovery: Treat a sustained AllReplicaBreakersOpen or a high steady-state trip rate as a capacity signal, not just a routing event — add replica capacity or tune parallel apply workers rather than widening the lag budget to silence the breaker.

In This Topic Area

These guides implement each layer of the pattern described above.

Implementing a Replica Health Circuit Breaker in Application Code A concrete, thread-safe per-replica breaker that checks lag before routing, trips on N consecutive breaches, and half-open probes — with the metrics wiring to make every transition observable.

Automatic Replica Ejection and Reintroduction on Lag Spikes Doing the same ejection at the infrastructure layer: HAProxy agent-checks returning drain/down, ProxySQL shunning, and Consul health, with hysteresis and warm-up to avoid flapping on reintroduction.

Graceful Degradation When All Replicas Exceed the Lag Budget What the global breaker does when no replica is eligible: primary fallback with load shedding, serving cached or stale-with-warning reads, and protecting the primary from a read stampede.

FAQ

Related

← Back to Replication Lag & Consistency Management

• Implementing a Replica Health Circuit Breaker in Application Code — the concrete, thread-safe breaker implementation with lag checks and metrics.
• Automatic Replica Ejection and Reintroduction on Lag Spikes — the same pattern enforced by HAProxy, ProxySQL, and Consul with anti-flap hysteresis.
• Graceful Degradation When All Replicas Exceed the Lag Budget — what the global breaker degrades to when no replica is eligible.
• Fallback Strategies When Replicas Fall Behind — the broader degraded-state playbook the breaker plugs into.
• Detecting and Handling Replication Lag in Real-Time — the lag telemetry that feeds every trip condition described here.