Automatic Replica Ejection and Reintroduction on Lag Spikes

Problem statement: A replica falls behind during a bulk import and your proxy keeps load-balancing reads to it because its TCP port is still open — you need the pool itself to eject a lagging node automatically and only bring it back once it has genuinely caught up, without flapping.

Symptom Identification

Ejection belongs at the routing tier, not in application code, when these patterns appear:

• A replica passes the proxy’s default TCP health check the entire time it is 30 seconds behind — the check only proves the port answers, not that the data is fresh.
• During a nightly batch job one replica’s lag climbs to minutes while the load balancer keeps sending it an equal share of reads, so a predictable fraction of users see stale data on a schedule.
• After the batch finishes, the replica catches up and the proxy immediately restores full weight; the sudden read surge on cold caches pushes it right back over the budget, and it oscillates in and out of the pool for several minutes.
• Multiple application fleets (web, mobile API, internal tools) all read through the same replicas, and you do not want to implement lag-aware ejection separately in each codebase.

The last point is the decisive one: when many clients share a pool, ejection at the proxy is enforced once for everyone, unlike the per-process breaker described in implementing a replica health circuit breaker in application code.

Root Cause Analysis

Default health checks test liveness, not freshness. A TCP or SELECT 1 check confirms the replica accepts connections. It says nothing about pg_last_wal_replay_lsn() position or Seconds_Behind_Source. A node can be perfectly reachable and hopelessly stale, and a liveness-only check keeps it in rotation. Ejection requires a freshness-aware check that reports lag to the proxy.

Reintroduction on a single sample causes flapping. Even with a lag-aware check, if the proxy re-admits the replica the instant one sample dips under the threshold, a node hovering at the budget will bounce. The fix is hysteresis — an eject threshold higher than the re-entry threshold — combined with a requirement that the re-entry condition hold for several consecutive checks. This is the same anti-flap principle the parent guide on circuit breaker patterns for degraded replicas applies to the CLOSED/OPEN transition, here expressed in proxy counters.

Cold reintroduction re-triggers the spike. A freshly caught-up replica has empty shared buffers and OS page cache. Hit it with a full traffic share and its query latency and apply lag both jump, re-crossing the budget. Reintroduction must be gradual.

Architecture

The lifecycle of a replica through a lag spike traces a clear timeline: eject on the way up, gate and warm on the way down. The hysteresis band between the two thresholds is what prevents oscillation.

Step-by-Step Resolution

Step 1: Write a lag-reporting agent

HAProxy’s agent-check polls a plain-text agent on the replica host that returns a weight and a state keyword. The agent measures lag and translates it into up, drain, or down.

# lag_agent.py — bound to agent-port 9101 on each replica host
import socketserver, psycopg2

EJECT_MS, REENTER_MS = 500, 200

def lag_ms():
    with psycopg2.connect("dbname=app host=127.0.0.1") as c, c.cursor() as cur:
        cur.execute("SELECT EXTRACT(EPOCH FROM "
                    "(now() - pg_last_xact_replay_timestamp()))*1000")
        return cur.fetchone()[0] or 0

class Handler(socketserver.BaseRequestHandler):
    def handle(self):
        try:
            lag = lag_ms()
        except Exception:
            self.request.sendall(b"down\n"); return
        if lag > EJECT_MS:
            self.request.sendall(b"drain\n")          # stop new, keep in-flight
        else:
            self.request.sendall(b"up 100%\n")        # full weight when fresh

Inline verification: printf "" | nc replica_a 9101 returns up 100% on a fresh replica and drain once lag exceeds EJECT_MS.

Step 2: Configure ejection with fall/rise counters

The proxy consumes the agent verdict but does not act on a single sample — fall and rise give the consecutive-check hysteresis.

backend pg_read
    balance leastconn
    option pgsql-check user haproxy_check
    default-server inter 2s fall 3 rise 5 slowstart 40s on-marked-down shutdown-sessions
    server replica_a 10.0.1.11:5432 check agent-check agent-addr 10.0.1.11 agent-port 9101 weight 100
    server replica_b 10.0.1.12:5432 check agent-check agent-addr 10.0.1.12 agent-port 9101 weight 100
    server replica_c 10.0.1.13:5432 check agent-check agent-addr 10.0.1.13 agent-port 9101 weight 100

fall 3 requires three consecutive breaching checks (6 seconds at inter 2s) before ejection, filtering out a single-scrape lag blip. The pgsql-check remains as the liveness backstop so a crashed node is marked down immediately regardless of the agent.

Inline verification: echo "show servers state pg_read" | socat stdio /var/run/haproxy.sock shows replica_a transition to a drained state after three breaching agent replies.

Step 3: Gate reintroduction on sustained catch-up

Reintroduction is the mirror image: rise 5 requires five consecutive healthy checks (10 seconds) before the drained server is eligible again. Because the agent reports up only under EJECT_MS, tighten reintroduction by lowering the agent’s return threshold to REENTER_MS while it is in a drained state — a small amount of state in the agent gives you true asymmetric hysteresis:

# stateful variant: only report 'up' once lag is under the lower re-entry mark
DRAINED = {"a": False}
def verdict(name, lag):
    if lag > EJECT_MS:
        DRAINED[name] = True
        return b"drain\n"
    if DRAINED[name] and lag > REENTER_MS:
        return b"drain\n"                     # still draining until under REENTER_MS
    DRAINED[name] = False
    return b"up 100%\n"

Inline verification: after a spike, the agent keeps returning drain while lag sits between REENTER_MS and EJECT_MS, and only returns up once lag drops below REENTER_MS.

Step 4: Warm up with slowstart

slowstart 40s in the default-server line tells HAProxy to ramp the reintroduced server’s effective weight linearly from near zero to full over 40 seconds. The node receives a trickle of reads first, warming shared buffers and OS page cache before it carries a full share.

    default-server inter 2s fall 3 rise 5 slowstart 40s

For a longer warm-up on large working sets, raise slowstart to 120s. Confirm the ramp is active by watching per-server request rate climb gradually rather than stepping to full.

Inline verification: in the HAProxy stats page, the reintroduced server’s weight field shows an intermediate value (e.g. 40) during the ramp, not an immediate 100.

Configuration Snippet

ProxySQL equivalent (MySQL)

ProxySQL shuns a replica automatically when its monitored lag exceeds max_replication_lag, and reintroduces it when lag recovers:

-- Set the per-host lag budget; ProxySQL shuns the node above this
UPDATE mysql_servers
   SET max_replication_lag = 5, max_connections = 400
 WHERE hostgroup_id = 20;

-- How often the monitor samples SHOW REPLICA STATUS
UPDATE global_variables SET variable_value = 1500
 WHERE variable_name = 'mysql-monitor_replication_lag_interval';

LOAD MYSQL SERVERS TO RUNTIME;
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

-- Observe the shunned state during a spike
SELECT hostgroup, srv_host, status, ConnUsed
  FROM stats_mysql_connection_pool WHERE hostgroup = 20;
-- status flips to SHUNNED_REPLICATION_LAG above the budget

Consul health-check alternative

A Consul service check that returns a critical status on lag ejects the replica from Consul-driven service discovery, which any Consul-Template-rendered proxy config then picks up:

{
  "service": {
    "name": "pg-read",
    "port": 5432,
    "checks": [{
      "args": ["/usr/local/bin/check_lag.sh", "--eject-ms", "500", "--reenter-ms", "200"],
      "interval": "2s",
      "success_before_passing": 5,
      "failures_before_critical": 3
    }]
  }
}

success_before_passing and failures_before_critical are Consul’s native hysteresis counters — the direct analogue of HAProxy rise/fall. This complements the broader connection routing and pooling strategy and the proxy-layer approach in read/write splitting at the proxy layer.

Verification and Rollback

Force a spike and watch the full cycle:

# Pause replay to drive lag up past the eject threshold
psql -h replica_a -c "SELECT pg_wal_replay_pause();"
sleep 10
echo "show servers state pg_read" | socat stdio /var/run/haproxy.sock   # expect drained

# Resume; lag drains, then gated + warm reintroduction
psql -h replica_a -c "SELECT pg_wal_replay_resume();"
watch -n2 'echo "show stat" | socat stdio /var/run/haproxy.sock | grep replica_a'

To roll back to liveness-only routing, remove the agent-check clause and keep only check (the pgsql-check). The pool reverts to accepting any reachable replica, and no node is ejected on lag — appropriate as a temporary measure if the agent itself is suspected of causing spurious ejections.

Edge Cases and Gotchas

The agent shares the fate of the node it measures

If the agent runs on the replica host and the host’s I/O saturates, the agent may itself become slow or unresponsive — HAProxy then marks the node down for the wrong reason, or misses a real lag spike because the agent timed out. Give the agent a short connection timeout to the local database and treat an agent timeout as drain, not up, so failure is fail-safe.

Ejecting the last healthy replica

Hysteresis and fall counters protect against flapping but not against ejecting everyone. If a fleet-wide event pushes all replicas over the budget, naive ejection empties the read pool entirely. Configure a floor — HAProxy does not eject below a minimum number of active servers if you set it up with a backup tier — or let the empty pool trigger the primary-fallback path described in graceful degradation when all replicas exceed the lag budget.

Draining does not cancel long-running reads

drain stops new sessions but lets in-flight queries run to completion, which is usually what you want. But a reporting query that started before the spike can keep running on the lagging node for minutes, reading increasingly stale data. Pair drain with a statement_timeout on the read role so no single query outlives the ejection by too long.

FAQ

Related

← Back to Circuit Breaker Patterns for Degraded Replicas

• Implementing a Replica Health Circuit Breaker in Application Code — the in-process equivalent for when you cannot rely on a shared proxy.
• Graceful Degradation When All Replicas Exceed the Lag Budget — what to do when ejection would empty the entire read pool.
• Detecting and Handling Replication Lag in Real-Time — the lag measurement the agent and ProxySQL monitor rely on.
• Implementing Read/Write Splitting at the Proxy Layer — how ejection fits into proxy-level read routing overall.