Replica Failover & Promotion Mechanics

Promotion is the one operation where a read replica stops being a read replica. Everything else in replication fundamentals and architecture — topology choice, commit mode, lag budgets — is about steady state; promotion is about the seconds in which steady state does not exist. The operational problem this page addresses is that those seconds are where the two worst outcomes in a replicated system live: lost transactions, when you promote a standby that had not received the tail of the WAL, and split brain, when the old primary keeps accepting writes after the new one has started. Both are silent at the moment they happen and expensive whenever they are discovered.

What Promotion Actually Is

Promotion is three distinct things that are easy to conflate.

The state change on one node. A standby in recovery stops replaying WAL from an external source and starts generating its own. In PostgreSQL this is pg_promote() or the promote action of pg_ctl; the node exits recovery, pg_is_in_recovery() starts returning false, and it begins accepting writes.

The timeline increment. At promotion the node increments its timeline ID and writes a .history file recording the WAL position at which the split occurred. From that moment, WAL position 0/5A000000 on timeline 7 and the same position on timeline 6 are different records. This is the mechanism that makes divergence detectable rather than silently corrupting — but it is only a detection mechanism, not a repair.

The topology change. Everything else in the system has to learn that a different address is now the write endpoint: the routing tier, the remaining standbys, the backup jobs, the monitoring. This is the part that takes the longest and fails most often.

Scope matters here. This page covers unplanned failover (the primary is gone or unreachable) and planned switchover (you are moving the primary deliberately, and the old primary is still healthy enough to cooperate). Switchover is strictly easier because the old primary can be shut down cleanly first, which eliminates the fencing problem entirely. Whenever you have the choice, convert an unplanned failover into a planned switchover by doing it before the node dies — during a maintenance window rather than during an incident.

Promotion is also distinct from rejoin, which is what the surviving standbys have to do afterwards, and from rebuild, which is what you do when rejoin is not possible. Most of the operational cost of a failover lands in those two steps, not in the promotion itself.

Timeline divergence at the promotion point A single line of WAL history runs left to right on timeline six. At the promotion point the promoted standby begins timeline seven and continues writing new records. A second standby had already received WAL past the promotion point on timeline six; those records exist on no surviving node, so the standby is ahead of the split on a dead branch and must be rewound back to the divergence point before it can follow timeline seven. WAL position → timeline 6 — shared history promotion point recorded in the .history file timeline 7 — new primary every new commit lands here timeline 6 — orphaned tail WAL the lagging standby already received but that exists on no surviving node pg_rewind: back to the split A standby past the split point cannot simply follow the new primary — its history contradicts the surviving one.

Mechanism Deep-Dive: The Five Steps in Order

The order below is not stylistic. Each step exists to make the next one safe, and reordering any two of them is the root cause of a recognisable class of incident.

Step 1 — Fence before you promote

Fencing means guaranteeing the old primary cannot accept another write. Until that guarantee exists, promoting a standby creates two writable nodes. The three mechanisms, in decreasing order of reliability:

  • Lease expiry. The primary holds a time-bounded lease in a consensus store (etcd, Consul, ZooKeeper) and demotes itself when it cannot renew. This is what Patroni does, and it is self-fencing: a partitioned primary removes itself without anyone needing to reach it.
  • STONITH. An external actor powers off or network-isolates the node. Reliable when it works, but it requires an out-of-band path that is itself frequently the thing that has failed.
  • Proxy-level blocking. The routing tier stops forwarding writes to the old address. This is the weakest form because it only fences clients that go through the proxy; anything with a direct connection string bypasses it entirely.
python
# The invariant every fencing mechanism must satisfy, stated as an assertion.
# lease_ttl is how long the old primary may still believe it is primary;
# detect_after is how long the failover controller waits before promoting.
assert detect_after > lease_ttl, (
    "promotion may begin only after the old primary's lease has certainly "
    "expired — otherwise both nodes accept writes for (lease_ttl - detect_after) seconds"
)

That single inequality is the whole of split-brain safety in a lease-based system. Patroni enforces it by construction; a hand-rolled failover script usually does not, which is why hand-rolled failover scripts produce split brain.

Step 2 — Choose the most advanced candidate

Not all standbys are equally current. Promoting an arbitrary one discards every transaction it had not received. Compare candidates on received LSN (what arrived over the wire) rather than replayed LSN (what has been applied), because a standby can be told to replay its pending WAL before promoting.

sql
-- Run on each candidate standby: what has arrived, and what has been applied.
SELECT pg_last_wal_receive_lsn()  AS received,
       pg_last_wal_replay_lsn()   AS replayed,
       pg_last_wal_receive_lsn() - pg_last_wal_replay_lsn() AS replay_backlog_bytes;

The candidate with the highest received LSN loses the fewest transactions. If a synchronous standby exists and was in sync at the moment of failure, it is by definition the correct candidate and the data loss is zero — this is the entire practical payoff of synchronous replication, and it only materialises if the failover controller actually prefers the synchronous node.

Step 3 — Promote and capture the new timeline

bash
# Promote, then wait for recovery to end rather than assuming it did.
psql -h standby-b -c "SELECT pg_promote(wait => true, wait_seconds => 60);"
psql -h standby-b -c "SELECT pg_is_in_recovery();"          # expect: f

# The new timeline ID — every surviving standby needs to be pointed at it.
psql -h standby-b -c "SELECT timeline_id FROM pg_control_checkpoint();"

pg_promote(wait => true) blocks until the node has genuinely left recovery. Scripts that fire promotion and immediately continue are the reason a routing tier sometimes gets repointed to a node that is still in recovery and therefore still read-only.

Step 4 — Repoint the routing tier

Until this step completes, the promotion has achieved nothing user-visible. The mechanism depends on how discovery works in your stack, and the choice made there dominates total failover time far more than anything the database does — the trade-offs are laid out in connection routing and pooling strategies. Verify the repoint by asserting from the client side that writes now land on the new node, not by reading the proxy’s own config.

Step 5 — Rejoin or rebuild the survivors

Every remaining standby was following the old primary on the old timeline. Each is now in one of two states: behind the split point, in which case it can simply be pointed at the new primary and will follow timeline 7 forward; or past the split point, holding orphaned WAL, in which case it must be rewound with pg_rewind or rebuilt from a base backup. Determining which state a standby is in is a comparison between its last received LSN and the switchpoint recorded in the new timeline’s history file.

Trade-off Comparison: Failover Controllers

Controller Fencing model Typical detect → serve Cascading-replica repair Operational cost
Patroni + etcd Self-fencing lease; primary demotes when it cannot renew 10–30 s Automatic pg_rewind on rejoin, configurable High setup, low steady-state; needs a healthy consensus store
repmgr + repmgrd Advisory; relies on witness quorum and optional external fencing 20–60 s repmgr standby follow, manual rewind decision Moderate; simpler to reason about, weaker split-brain guarantee
Managed RDS / Aurora Provider-internal, not observable 30–120 s (RDS Multi-AZ); seconds (Aurora) Provider handles it; replicas may restart Lowest; least control and least visibility
Manual runbook Whatever the operator does Minutes Manual, per node Lowest tooling cost, highest incident cost and variance
Proxy-only (no controller) Proxy stops routing; old primary still writable Fast to route, unsafe to promote None Deceptively cheap — do not promote from a proxy health check alone

The last row deserves emphasis. A proxy health check is a good input to failover and a terrible trigger for it: the proxy can only observe reachability from itself, so a proxy-side network fault looks identical to a dead primary. Promotion decisions need a quorum of observers, which is precisely what a consensus store provides. The trade-offs between provider-managed and self-managed control are explored further in managed vs self-managed read replicas.

Where failover time is spent, by discovery mechanism Two stacked timelines on a common seconds axis. With a consensus store and a proxy repoint, detection takes most of the budget, fencing and promotion are brief, and routing converges quickly, for roughly twenty seconds end to end. With DNS-based discovery the same detection and promotion phases are followed by a routing convergence phase several times longer than everything before it, dominated by resolver cache expiry. 0 s 30 s 60 s 90 s consensus store + proxy repoint detect ≈ 28 s to first successful write same cluster, DNS discovery detect routing convergence — resolver caches expire ≈ 80 s to first successful write detection fencing promotion routing convergence Promotion is the smallest slice in both cases. Optimising the database rarely shortens a failover; optimising discovery always does. Standby rejoin happens after service is restored and does not extend the outage — unless it empties the read pool.

Configuration Runbook

Patroni: the settings that decide your failover behaviour

yaml
# patroni.yml — the fields that actually change failover outcomes
bootstrap:
  dcs:
    ttl: 30                      # lease lifetime: the old primary self-demotes within this
    loop_wait: 10                # how often each node re-evaluates cluster state
    retry_timeout: 10            # DCS call timeout; ttl must exceed loop_wait + 2*retry_timeout
    maximum_lag_on_failover: 1048576   # bytes — refuse to promote a standby further behind than this
    synchronous_mode: true             # only a synchronous standby may be promoted
    synchronous_mode_strict: false     # false: allow writes if no sync standby remains (availability
                                       # over durability). true: block writes instead. Pick deliberately.
postgresql:
  use_pg_rewind: true            # rejoin diverged standbys automatically instead of rebuilding
  parameters:
    wal_log_hints: on            # REQUIRED for pg_rewind; enabling it later needs a restart

Three of these deserve a second look. maximum_lag_on_failover is the durability floor: it is the largest amount of data you are willing to lose to restore service, expressed in bytes of WAL. synchronous_mode_strict is the availability-versus-durability switch, and it has no safe default — true means an outage rather than data loss, false means the reverse. And wal_log_hints must be on before the incident; discovering it is off while trying to rewind a diverged standby means rebuilding from a base backup instead.

repmgr: promotion and follow

bash
# On the chosen standby — dry run first, always.
repmgr -f /etc/repmgr.conf standby promote --dry-run
repmgr -f /etc/repmgr.conf standby promote

# On every surviving standby — repoint at the new primary.
repmgr -f /etc/repmgr.conf standby follow --upstream-node-id=3

# If follow fails with a divergence error, rewind first:
pg_rewind --target-pgdata=/var/lib/postgresql/16/main \
          --source-server="host=new-primary user=repmgr dbname=repmgr" \
          --progress

repmgr standby follow will refuse rather than silently corrupt when the standby has diverged, which is the correct behaviour — the refusal is the signal to rewind.

Managed RDS: what you can and cannot control

On RDS you do not run any of the above. What you control is the shape of the topology: whether a Multi-AZ standby exists, how many read replicas there are, and where they sit. What you cannot control is the promotion decision or its timing. Two consequences follow. First, a Multi-AZ failover replaces the primary endpoint’s DNS target, so your clients’ DNS caching becomes the dominant recovery variable — see cross-region read replicas on RDS: cost and lag trade-offs for how this interacts with cross-region topologies. Second, read replicas of a Multi-AZ instance may briefly disconnect and reconnect during the failover, so your read tier must tolerate a gap it does not control.

Monitoring and Alerting Signals

Failover monitoring has an unusual property: the signals that matter most are the ones that tell you a failover would go badly if it happened now. Alert on those in steady state rather than waiting for the incident.

promql
# 1. Promotion readiness: is any standby close enough to promote without data loss?
min by (cluster) (pg_replication_lag_bytes) > 1048576

# 2. Consensus store health — no quorum means no safe promotion
up{job="etcd"} < 1

# 3. Split-brain canary: more than one node reporting itself as primary
count by (cluster) (pg_is_in_recovery == 0) > 1

# 4. Timeline drift — a standby on a different timeline than the primary
count by (cluster) (count by (cluster, timeline_id) (pg_control_timeline)) > 1

# 5. Rewind capability: wal_log_hints off means rebuild, not rejoin
pg_settings_wal_log_hints == 0

Signal 3 is the one to page on unconditionally and at any hour. Two primaries is not a degradation, it is a correctness failure that gets worse every second it continues. Signals 2 and 5 are steady-state readiness checks and belong on a dashboard rather than a pager, but they should be reviewed before any planned maintenance. The wider metric vocabulary these build on is covered in Prometheus metrics for replica health and lag.

Failure Modes and Recovery Steps

  1. Split brain — two writable primaries. Diagnosis: signal 3 above fires, or clients report writes that later vanish. Recovery: immediately fence the node with fewer accepted writes (usually the old primary), then reconcile. Reconciliation is manual: extract the divergent transactions from the losing node’s WAL with pg_waldump or logical decoding and re-apply them against the winner. There is no automatic merge. Prevention: the lease inequality in step 1.

  2. Promoted a standby that was too far behind. Diagnosis: application reports missing recent writes immediately after failover; the promoted node’s pre-promotion received LSN was well behind the old primary’s flush LSN. Recovery: if the old primary’s disk survives, the missing WAL is recoverable from it — archive that WAL before doing anything else, because a rebuild will destroy it. Prevention: set maximum_lag_on_failover and prefer synchronous standbys.

  3. Surviving standbys cannot rejoin. Diagnosis: standby follow or Patroni rejoin fails with a divergence error; pg_rewind reports it cannot find a common checkpoint. Recovery: rebuild the standby from a fresh base backup (pg_basebackup -R) — this is slow but always works. Prevention: wal_log_hints = on and enough wal_keep_size or a replication slot to cover the rewind window.

  4. Read pool empties during the failover. Diagnosis: read latency collapses and the primary’s connection count spikes at the moment of promotion. Recovery: nothing fast — this one is prevented, not fixed. Prevention: size the read tier so that losing the promotion candidate leaves capacity, and pre-decide the empty-pool policy described in fallback strategies when replicas fall behind.

  5. Cascading replicas orphaned. Diagnosis: in a cascading topology, replicas downstream of the promoted node keep streaming happily while replicas downstream of a non-promoted intermediate stall silently. Recovery: repoint each downstream node at a surviving upstream, working from the new primary outward. Prevention: record the cascade parentage somewhere the failover controller can read, because Patroni and repmgr both handle the first level well and deeper levels poorly.

Production-Readiness Checklist

In This Section

FAQ

What is a timeline and why does promotion create a new one?

A timeline is PostgreSQL’s identifier for a distinct line of WAL history. Promotion increments it so that WAL written by the newly promoted primary can never be confused with WAL written by the old one at the same log position. This is what makes divergence detectable: a standby that received WAL from the old primary beyond the promotion point holds records that no longer exist on the new timeline, and it must be rewound or rebuilt before it can follow the new primary.

Should automated failover be enabled in production?

Enable it only when you also have reliable fencing and a quorum that cannot be satisfied by a partitioned minority. Automated failover without fencing converts a network partition into two writable primaries, which is far more damaging than the outage it was meant to prevent. If you cannot fence, prefer automated detection with a human-confirmed promotion step.

Do read replicas keep serving reads during a failover?

The promoted node stops serving as a replica the moment it is promoted, and any other standby that was replicating from the old primary stalls once that primary is gone. So read capacity drops by at least one node and often by the whole tier until the standbys are repointed. Size the read pool so that losing the promotion candidate plus one more node still leaves enough capacity, and decide in advance whether reads fall back to the new primary during the gap.

How long should promotion take?

The promotion itself is fast — typically under a second once the standby has replayed its pending WAL. The wall-clock time your users experience is dominated by detection, fencing, and routing convergence. In a tuned Patroni cluster the whole sequence is commonly 10 to 30 seconds; DNS-based routing routinely adds a minute or more on top, which is why the routing tier, not the database, is usually the thing to optimise.

← Back to Database Replication Fundamentals & Architecture