Preventing Split-Brain During Automated Replica Promotion

Problem statement: automated failover promoted a standby while the old primary was merely unreachable rather than dead, and for some number of seconds two nodes accepted writes — producing two divergent histories that no tool can merge.

Split-brain is the worst outcome available to a replicated database, worse than the outage automation was installed to prevent, because an outage ends and divergent data does not. This page is about the specific mechanisms that make it impossible rather than unlikely. The surrounding operation is described in replica failover and promotion mechanics.

Symptom Identification

Split-brain in progress, or recently past:

  • Two nodes simultaneously return false from pg_is_in_recovery().
  • Application logs show writes succeeding that later cannot be found — a record created, then a subsequent read returning nothing, with no deletion in between.
  • Two nodes report the same timeline ID but different pg_current_wal_lsn() values that both keep advancing.
  • Sequence values collide: two rows with the same primary key from different nodes, discovered when the nodes are reconciled.
  • The consensus store shows a leader key held by one node while another node’s log says it promoted itself.

The configurations that make it possible, before it has happened:

  • Automatic failover enabled with no fencing action in the promotion hook.
  • A two-node consensus store, or a consensus store co-located with the primary.
  • A failover trigger derived from a single observer — one proxy, one monitoring host, one health-check endpoint.
  • Lease timing where ttl ≤ loop_wait + 2 × retry_timeout.

Root Cause Analysis

Split-brain is always the same bug: the promotion decision was made by observers who could not distinguish “dead” from “unreachable”.

A network partition produces exactly this ambiguity. The primary is alive and serving whichever clients share its side of the partition. The standbys and the monitoring see nothing. From their side the primary is indistinguishable from a crashed one, and any logic that promotes on “primary not visible” will promote.

Two mechanisms resolve the ambiguity, and they work in opposite directions.

Quorum resolves it from the outside. If promotion requires agreement from a strict majority of a fixed membership, then a minority partition mathematically cannot promote — there is no configuration of an isolated minority that reaches majority. This does not stop the old primary from writing; it stops a second primary from appearing.

Self-fencing resolves it from the inside. If the primary holds a time-bounded lease and demotes itself when it cannot renew, then a partitioned primary removes itself without anyone needing to reach it. This is the stronger property, because it does not depend on anyone else acting, and it is why lease-based systems are safer than vote-based ones.

Used together they close the window completely: the minority cannot elect, and the old primary stops on its own. Used separately, each leaves a gap. Quorum alone permits the old primary to keep writing on its side until something else stops it. Self-fencing alone, without quorum, permits two isolated groups to each elect a leader.

The timing inequality is what makes self-fencing sound. Demotion must be guaranteed to complete before promotion may begin:

text
ttl  >  loop_wait + 2 × retry_timeout

The left side is how long the lease lives. The right side is the longest a primary can spend discovering it has lost contact. If the right side can exceed the left, a primary can still believe itself leader after the lease has expired and a new one has been elected. Every lease-based system has this inequality in some form, and every hand-rolled failover script omits it.

Quorum and self-fencing across a network partition A partition line divides the deployment. On the minority side sit the old primary and one consensus node; they hold one vote of three and cannot reach majority, so no leader can be elected there, and the old primary demotes itself when its lease expires. On the majority side sit two consensus nodes and two standbys; they hold two votes of three, reach majority, and elect a new leader. Clients on the minority side lose write access, which is the correct outcome. network partition minority side — 1 of 3 votes old primary lease expires → demotes etcd node 1 cannot reach majority clients here lose writes — correct no election possible here, by arithmetic, not by policy majority side — 2 of 3 votes etcd node 2 etcd node 3 quorum reached standby → new primary promoted after lease expiry standby Two independent guarantees hold here simultaneously: 1. the minority cannot elect — quorum arithmetic 2. the old primary stops on its own — lease expiry Either alone leaves a window. Both together leave none.

Step-by-Step Resolution

Step 1 — Size the consensus store so a minority can never win

bash
# Membership and health. Three members across three failure domains.
etcdctl member list -w table
etcdctl endpoint status --cluster -w table

Three members tolerate the loss of one; five tolerate two. What matters more than the count is the placement: if two of three members share a zone with the primary, a single zone outage takes the primary and quorum together, and no promotion is possible anywhere.

Inline verification: stop one member and confirm etcdctl endpoint health --cluster still reports the etcd cluster healthy and writable; stop two of three and confirm writes now fail — that failure is the property protecting you.

Step 2 — Make the primary self-fence

bash
patronictl -c /etc/patroni.yml show-config | grep -E '^ttl|loop_wait|retry_timeout'

Patroni self-fences by construction: a leader that cannot write its key to the consensus store within ttl demotes itself to a read-only standby. Nothing needs to reach it. If you are not using a lease-based controller, this property must be built — most commonly as a watchdog that stops PostgreSQL when a heartbeat to the store fails.

Inline verification: block the primary’s access to the consensus store with a firewall rule and confirm that within ttl seconds pg_is_in_recovery() on that node returns true without anyone having intervened.

Step 3 — Check the timing inequality

bash
python3 - <<'PY'
ttl, loop_wait, retry_timeout = 40, 10, 7
margin = ttl - (loop_wait + 2 * retry_timeout)
print(f"margin = {margin}s", "OK" if margin > 0 else "UNSAFE — two writable nodes possible")
PY

A zero or negative margin means the configuration permits a window in which the old primary still believes it is leader while a new one exists. Widening ttl slows failover slightly and is always the right trade against the alternative.

Inline verification: the script prints a positive margin; record the number in the runbook so a future change to retry_timeout is checked against it.

Step 4 — Add network-level fencing as the second layer

Self-fencing handles the well-behaved case. A node whose process is wedged — long GC pause, frozen VM, I/O hang — may resume after the lease expired and issue writes it prepared beforehand. Network fencing closes that:

bash
#!/usr/bin/env bash
# /usr/local/bin/fence-old-primary.sh — invoked before promotion completes.
# Revoke client reachability. The node may still be running; it can no longer
# be reached by anything that would write to it.
set -euo pipefail
OLD_PRIMARY_ID="${1:?instance id required}"

aws ec2 modify-instance-attribute \
    --instance-id "$OLD_PRIMARY_ID" \
    --groups "$FENCED_SECURITY_GROUP"      # a group with no ingress on 5432

# Confirm the fence took effect before returning success — a fencing script
# that reports success without verifying is worse than none, because the
# controller then proceeds believing the node is contained.
for _ in $(seq 1 10); do
  if ! timeout 2 bash -c "</dev/tcp/${OLD_PRIMARY_HOST}/5432" 2>/dev/null; then
    echo "fenced"; exit 0
  fi
  sleep 1
done
echo "FENCE FAILED — refusing to promote" >&2
exit 1

The non-zero exit on failure is the important line. A fencing hook that always succeeds is decoration.

Inline verification: run the script against a test instance and confirm a subsequent psql -h old-primary times out.

Step 5 — Deploy the split-brain canary

yaml
# The one alert that should page unconditionally, at any hour.
groups:
  - name: split-brain
    rules:
      - alert: MultiplePrimariesDetected
        expr: count by (cluster) (pg_is_in_recovery == 0) > 1
        for: 0m                     # no 'for' delay — every second matters
        labels: {severity: critical}
        annotations:
          summary: "{{ $value }} nodes report themselves primary in {{ $labels.cluster }}"
          action: "Stop writes to the node with fewer accepted transactions. Preserve its WAL before any rebuild."

Inline verification: during a drill, promote a second node deliberately in an isolated environment and confirm the alert fires within one scrape interval.

Configuration Snippet

yaml
# The complete set of settings that determine whether split-brain is possible.
bootstrap:
  dcs:
    # --- Layer 1: quorum ---------------------------------------------------
    # Provided by the etcd/Consul cluster itself: an odd membership spread
    # across failure domains. Patroni inherits the guarantee; it does not
    # create it. A single-node etcd gives you NO quorum protection at all.

    # --- Layer 2: self-fencing lease ---------------------------------------
    ttl: 40                # lease lifetime — the primary demotes itself at expiry
    loop_wait: 10          # renewal interval
    retry_timeout: 7       # per-attempt DCS timeout
    # Safety condition: 40 > 10 + 2×7 = 24. Margin 16 s.

    # --- Layer 3: candidate constraints ------------------------------------
    synchronous_mode: true
    maximum_lag_on_failover: 1048576

postgresql:
  # --- Layer 4: hardware/OS watchdog ---------------------------------------
  # If the process stalls past the lease, the watchdog resets the machine.
  # This is the only layer that survives a frozen VM resuming with queued writes.
  use_unix_socket: true
  callbacks:
    on_role_change: /usr/local/bin/patroni-repoint.sh

watchdog:
  mode: required           # refuse to run as leader without a working watchdog
  device: /dev/watchdog
  safety_margin: 5         # ping the watchdog until ttl-5; then let it fire

watchdog: mode: required is the setting that turns “we believe the primary stops” into “the primary stops or the machine resets”. Patroni pings the hardware watchdog while it holds the leader lease and stops pinging safety_margin seconds before expiry; if the process is wedged and cannot demote cleanly, the kernel resets the box. It is the only layer that defends against a stalled process resuming after promotion, and it is off by default.

Four layers of split-brain defence and each layer's residual gap Four nested layers listed from outermost to innermost. Quorum stops a minority electing a leader but lets the old primary keep writing. A self-fencing lease stops a cooperative primary but not a stalled process. Network fencing stops a reachable primary but depends on an out-of-band control path. A hardware watchdog stops a wedged process by resetting the machine and is the only layer with no residual gap, at the cost of needing a working watchdog device. layer stops still lets through 1 · quorum odd membership, spread a second election old primary keeps writing 2 · self-fencing lease ttl > loop_wait + 2×retry a cooperative primary a stalled or frozen process 3 · network fencing revoke client reachability a reachable primary a broken control path 4 · hardware watchdog mode: required a wedged process nothing — needs a real device Layers 1 and 2 are the minimum for automated failover. Layer 4 is what makes it genuinely safe.

Verification and Rollback

Drill the partition rather than reasoning about it. The only convincing evidence is a test that isolates the primary and confirms nothing bad happens:

bash
# On the primary: sever it from the consensus store AND the standbys,
# while leaving a client path intact — this is the dangerous shape.
sudo iptables -A OUTPUT -d 10.0.3.0/24 -j DROP    # etcd subnet
sudo iptables -A OUTPUT -d 10.0.1.0/24 -j DROP    # standby subnet

# From a client that can still reach the old primary, attempt a write.
psql -h old-primary -c "INSERT INTO split_probe DEFAULT VALUES;"
# Expected within ttl seconds: ERROR: cannot execute INSERT in a read-only
# transaction — the node has demoted itself.

# Clean up.
sudo iptables -D OUTPUT -d 10.0.3.0/24 -j DROP
sudo iptables -D OUTPUT -d 10.0.1.0/24 -j DROP

If the write succeeds after ttl has elapsed, self-fencing is not working and automated failover is unsafe in this deployment. That is the single most valuable finding a drill can produce.

Confirm only one primary exists, continuously:

sql
-- Run against every node; exactly one row should return false.
SELECT inet_server_addr() AS node, pg_is_in_recovery() AS in_recovery;

Recovery when split-brain has already occurred. There is no rollback, only reconciliation, and the order is critical:

  1. Stop writes to one side. Choose the side with fewer accepted transactions — usually the isolated old primary.
  2. Preserve its WAL before anything else. tar the pg_wal directory and the data directory to durable storage. A rebuild destroys the only copy of the divergent transactions, and this step is irreversible if skipped.
  3. Extract the divergent transactions with pg_waldump from the switchpoint LSN forward, or with logical decoding if a slot exists.
  4. Reconcile against the surviving primary at the application level. There is no generic merge; only the application knows whether two conflicting rows should be combined, one preferred, or a human consulted.
  5. Rebuild the losing node from the survivor with pg_basebackup once the extraction is complete and verified.

Edge Cases and Gotchas

The consensus store co-located with the primary

Placing etcd members on the database hosts is convenient and removes the guarantee. A zone outage that takes the primary takes its etcd member too; with three members across three database hosts in two zones, losing the busier zone loses quorum, and no promotion can happen anywhere. Run the consensus store on separate, small instances distributed independently of the databases.

A two-node cluster cannot be made safe automatically

With two data nodes and no witness there is no majority to appeal to — each side sees one of two, and any rule that lets one side promote lets both. No configuration fixes this. Either add a third voter (a witness node costs very little) or set failover to manual and accept human confirmation as the tiebreaker.

Clients with direct connection strings bypass every proxy-level control

Fencing that operates only at the proxy leaves any service holding a direct DSN writing to the old primary. Audit for these before trusting proxy-level containment: SELECT DISTINCT client_addr, application_name FROM pg_stat_activity WHERE backend_type = 'client backend' on the primary shows who is connected and from where, and anything that is not the proxy’s address is outside your fence. This is one reason the routing tier’s design matters for correctness and not only for performance — see connection routing and pooling strategies.

FAQ

Why is a proxy health check not a fencing mechanism?

Because it only observes reachability from the proxy, and it only controls traffic that goes through the proxy. A network fault between the proxy and the primary looks exactly like a dead primary, and any client holding a direct connection string keeps writing regardless of what the proxy decides. Fencing must make the old primary stop accepting writes, not merely stop one path from reaching it.

How many consensus nodes do I need?

An odd number, at least three, spread across independent failure domains. Three tolerates one loss; five tolerates two. An even number gains you nothing over the odd number below it and creates the possibility of a tie. Crucially, the nodes must not share a failure domain with each other or with the database primary, or a single zone outage removes both your primary and your ability to safely replace it.

Can split-brain happen with synchronous replication enabled?

Yes. Synchronous replication constrains where data goes, not who may write. An isolated old primary with synchronous_standby_names cleared, or with a standby still reachable from its side of the partition, can continue committing while a new primary is elected on the other side. Synchronous replication reduces data loss at promotion; it does not fence.

What do I do if split-brain has already happened?

Stop writes to one side immediately — normally the one with fewer accepted transactions — and preserve its write-ahead log before doing anything else, because a rebuild destroys the only copy of the divergent transactions. Then reconcile by extracting those transactions with logical decoding or pg_waldump and replaying them against the surviving primary. There is no automatic merge; the reconciliation is application-specific and manual.

← Back to Replica Failover & Promotion Mechanics