repmgr Standby Promotion and Rejoin Runbook

Problem statement: the primary is gone, you are running repmgr rather than a consensus-backed controller, and you need the exact command sequence — including what to do at each point where a command refuses to proceed.

repmgr is deliberately conservative: several of its commands will stop and tell you why rather than doing something that might corrupt a node. Knowing which refusals are expected, and what each one means, is most of what separates a calm failover from a long one. The conceptual background is in replica failover and promotion mechanics.

Symptom Identification

You are in this runbook when:

  • repmgr cluster show reports the primary as ? unreachable or - failed, and one or more standbys as running.
  • Application logs show connection failures to the write endpoint while reads against standbys still succeed.
  • repmgrd has logged a failover decision but the topology has not converged — some standbys are following a node that no longer exists.
  • A previous promotion left nodes in running state but with upstream pointing at the old primary’s node ID.

Distinguish this from the case where the primary is reachable but degraded. If the primary still answers, you want a switchover (repmgr standby switchover), which is cleaner and cannot lose data because the old primary is shut down first.

Root Cause Analysis

repmgr’s model is a metadata table plus commands that act on it. Unlike a lease-based controller, there is no continuously-renewed token that the primary must hold. repmgr.nodes records who is upstream of whom; repmgrd, if running, watches connectivity and may act. Everything else is a command someone or something invokes.

Two consequences follow, and both shape the runbook.

First, there is no automatic fencing. Nothing in repmgr prevents an unreachable-but-alive primary from continuing to serve writes to clients that can still reach it. Quorum among visible nodes decides whether repmgrd promotes; it does not decide whether the old primary stops. Fencing, if you want it, has to be an explicit action in the promotion hook.

Second, the metadata can drift from reality. After a promotion, repmgr.nodes is updated on the new primary, but a standby that was unreachable at that moment may still believe its upstream is the old node. It will happily keep trying to stream from an address that is gone, appearing “up” to a naive health check while receiving nothing. Reconciling the metadata is a required step, not an optional tidy-up.

Three states a surviving standby can be in after promotion The new primary sits on the left. Three standbys are shown. The first received WAL only up to the divergence point, so standby follow succeeds directly. The second received WAL past the divergence point, so its history contradicts the new primary and standby follow refuses until node rejoin with a rewind. The third was unreachable during the promotion, so its metadata still names the dead primary as upstream and it is streaming from nothing while appearing healthy. new primary timeline 7 behind the split point standby follow succeeds as-is one command, no data movement past the split point follow refuses — divergence detected node rejoin --force-rewind was unreachable still names the dead primary as upstream looks healthy, receives nothing The third case is the dangerous one: no command failed, so nothing draws attention to it until the data is visibly old.

Step-by-Step Resolution

Step 1 — Confirm the failure from more than one vantage point

bash
# From a standby, not from your laptop — the question is what the cluster sees.
repmgr -f /etc/repmgr.conf cluster show
repmgr -f /etc/repmgr.conf cluster matrix   # every node's view of every other node

cluster matrix is the command that distinguishes “the primary is dead” from “I cannot reach the primary”. If the witness and the other standbys can all still reach it, you have a local network fault and promoting would create split brain.

Inline verification: in cluster matrix, the primary’s column shows ? from every row, not just from the node you are logged into.

Step 2 — Dry-run the promotion

bash
repmgr -f /etc/repmgr.conf standby promote --dry-run --log-level=DEBUG

The dry run reports which checks it would apply: whether the current primary is genuinely unreachable, whether this node is the most advanced standby, and whether any other node is already promoting. Read the output rather than skimming for the exit code — it frequently names a better candidate than the node you happened to log into.

Inline verification: the dry run ends with standby promotion would succeed and no WARNING lines about a more advanced sibling.

Step 3 — Promote

bash
repmgr -f /etc/repmgr.conf standby promote
psql -c "SELECT pg_is_in_recovery();"        # expect: f
psql -c "SELECT timeline_id FROM pg_control_checkpoint();"   # note this number

Record the timeline ID. Every subsequent step is about getting the other nodes onto it.

Inline verification: repmgr cluster show lists this node as primary * running.

Step 4 — Repoint each surviving standby

On each remaining standby, in turn:

bash
repmgr -f /etc/repmgr.conf standby follow --upstream-node-id=3 --dry-run
repmgr -f /etc/repmgr.conf standby follow --upstream-node-id=3

If this succeeds, the node was behind the split point and is now streaming from the new primary. If it fails with a divergence error, move to the rewind path:

bash
# The node must be STOPPED before rejoin — rejoining a running node is refused.
pg_ctl -D /var/lib/postgresql/16/main stop -m fast

repmgr -f /etc/repmgr.conf node rejoin \
  -d 'host=new-primary user=repmgr dbname=repmgr' \
  --force-rewind --config-files=postgresql.conf,pg_hba.conf --verbose

--config-files matters: pg_rewind copies the data directory contents from the source, which would otherwise overwrite this node’s local configuration with the new primary’s. Naming the files preserves them across the rewind.

Inline verification: on the new primary, SELECT application_name, state FROM pg_stat_replication; lists this node with state = streaming.

Step 5 — Reconcile the metadata

bash
# Every node's recorded upstream must match reality.
psql -d repmgr -c "SELECT node_id, node_name, type, upstream_node_id, active
                     FROM repmgr.nodes ORDER BY node_id;"

# Re-register anything whose row is wrong or missing.
repmgr -f /etc/repmgr.conf standby register --force --upstream-node-id=3

This is the step most often skipped, and it is the step that determines whether the next failover starts from an accurate picture.

Inline verification: every row’s upstream_node_id is either the new primary’s ID or NULL for the primary itself, and active is true for every node that is running.

Configuration Snippet

ini
# /etc/repmgr.conf — the settings that govern failover behaviour.
node_id=3
node_name='pg-node-3'
conninfo='host=pg-node-3 user=repmgr dbname=repmgr connect_timeout=2'
data_directory='/var/lib/postgresql/16/main'

# --- Failover behaviour -------------------------------------------------
failover=automatic                 # 'manual' if you have no fencing (see gotchas)
promote_command='repmgr standby promote -f /etc/repmgr.conf --log-to-file'
follow_command='repmgr standby follow -f /etc/repmgr.conf --log-to-file --upstream-node-id=%n'

# How long a node must be unreachable before repmgrd acts.
# reconnect_attempts × reconnect_interval is your detection window.
reconnect_attempts=6
reconnect_interval=5               # → 30 s before failover is considered

# Refuse to promote a standby further behind than this (bytes of WAL).
# The durability floor, same role as Patroni's maximum_lag_on_failover.
monitoring_history=yes

# --- Rejoin ------------------------------------------------------------
# Required for pg_rewind; must be enabled BEFORE the incident (needs a restart).
# Verify with: SHOW wal_log_hints;
pg_bindir='/usr/lib/postgresql/16/bin'

# Fire a fencing action before the promotion completes. Without this,
# repmgr does not stop the old primary from accepting writes.
service_promote_command='/usr/local/bin/fence-old-primary.sh && pg_ctl promote -D %d'

The service_promote_command hook is where fencing lives, and by default it does not exist. A minimal fencing script revokes the old primary’s network reachability from clients — a security-group change, an ACL update, or an instance stop — and must succeed before promotion proceeds. If it cannot succeed reliably, set failover=manual and accept a slower, supervised failover in exchange for not creating two writable primaries automatically.

What to do when standby follow refuses A decision path. Standby follow fails with a divergence error. First check whether write ahead log hints or data checksums were enabled: if not, rewind is impossible and the only route is a full rebuild from a base backup. If they were enabled, stop the node and run node rejoin with force rewind. If rejoin reports no common checkpoint, the divergence predates the retained write ahead log and a rebuild is again the answer. follow refuses divergence detected wal_log_hints was on? node rejoin --force-rewind yes common checkpoint? rebuild: pg_basebackup -R slow, always works, no prerequisites no no Enabling wal_log_hints needs a restart, so it can only be fixed before the incident — never during one.

Verification and Rollback

Confirm the topology converged:

bash
repmgr -f /etc/repmgr.conf cluster show
# Expect: exactly one 'primary * running'; every other node 'standby running'
# with its upstream column naming the new primary.

repmgr -f /etc/repmgr.conf cluster crosscheck
# Every cell should be '*'. A '?' means one node cannot see another and the
# next failover decision will be made on incomplete information.

Confirm streaming is real, not merely configured:

sql
-- On the new primary. Every standby should appear here, streaming, with
-- a replay_lag that is small and shrinking.
SELECT application_name, state, sync_state,
       pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replay_backlog_bytes,
       replay_lag
  FROM pg_stat_replication ORDER BY application_name;

A standby that is configured to follow but absent from pg_stat_replication is the third case from the diagram — it believes it is following and is not.

Rollback. If the promotion was premature — the old primary turns out to be healthy and more advanced — the safe reversal is a switchover back, not an attempt to un-promote:

bash
# From the old primary, once it has been rejoined as a standby:
repmgr -f /etc/repmgr.conf standby switchover --siblings-follow --dry-run
repmgr -f /etc/repmgr.conf standby switchover --siblings-follow

--siblings-follow moves the other standbys along with the switchover, which avoids leaving half the fleet pointing at the wrong node. Note the ordering constraint: the old primary must first be rejoined as a standby before it can be switched back to primary. There is no path that skips that step.

Edge Cases and Gotchas

node rejoin refuses because the node is running

pg_rewind operates on a cleanly shut down data directory. A node that was killed rather than stopped may need to be started and stopped once to complete crash recovery before rejoin will accept it:

bash
pg_ctl -D /var/lib/postgresql/16/main start     # let crash recovery finish
pg_ctl -D /var/lib/postgresql/16/main stop -m fast
repmgr -f /etc/repmgr.conf node rejoin -d '...' --force-rewind

Starting a node that was a primary means it briefly becomes writable again. Fence it at the network level first, or start it with hot_standby=off and a pg_hba.conf that rejects application users.

The witness must not live beside the primary

A witness node placed in the same rack, zone, or hypervisor as the primary provides no useful vote — the partition that isolates the primary isolates the witness with it, and the remaining standbys see two nodes vanish at once. Place the witness where the majority of your clients are, so its view approximates the users’ view.

repmgrd and manual commands can race

If repmgrd is running with failover=automatic and an operator simultaneously runs standby promote by hand, two promotions can be in flight. Before any manual intervention, pause the daemon:

bash
repmgr -f /etc/repmgr.conf service pause
# ... manual work ...
repmgr -f /etc/repmgr.conf service unpause

service pause is the repmgr equivalent of putting a replica set in maintenance mode, and skipping it is a reliable way to turn one incident into two.

FAQ

Why does standby follow fail after a promotion?

Because the standby has diverged: it received WAL from the old primary past the point where the new primary’s timeline split off, so its history contradicts the new primary’s. repmgr refuses rather than corrupting the node. The fix is node rejoin --force-rewind, or a full rebuild if no common checkpoint exists.

Do I need a witness node?

You need one whenever automatic failover is enabled and you have an even number of data nodes, because otherwise a network partition can leave each side with an equal vote and no way to distinguish a dead primary from an unreachable one. The witness is a lightweight PostgreSQL instance that holds no data and exists solely to break that tie, and it must sit where the majority of your clients are rather than alongside the primary.

Can repmgr rejoin a node that was running as a primary?

Yes, that is exactly what node rejoin is for. The old primary is shut down cleanly, rewound onto the new primary’s timeline, and restarted as a standby. It requires wal_log_hints or data checksums to have been enabled beforehand, and the node must be stopped — rejoining a running node is refused.

Is repmgrd’s automatic failover safe without external fencing?

Less safe than a lease-based system. repmgrd decides by quorum among visible nodes, but nothing physically stops an unreachable-yet-alive primary from continuing to accept writes from clients that can still reach it. If you enable automatic failover, pair it with a fencing action in service_promote_command, or accept that split brain is possible and monitor for it explicitly.

← Back to Replica Failover & Promotion Mechanics