Reconfiguring Cascading Replicas After a Primary Promotion

Problem statement: the promotion succeeded, the first-level standbys are following the new primary, and somewhere behind them a second or third level of replicas is quietly streaming from nothing — serving increasingly stale reads without any command having failed.

Cascading topologies pay for themselves on wide-area links, as described in designing multi-region read replica topologies. What they cost is that failover stops being a flat operation. This page is the ordered walk that repoints a tree, and the reason the order matters.

Symptom Identification

The characteristic sign is a node that looks healthy and is not receiving data:

  • pg_is_in_recovery() returns true, the process is running, connections succeed — but pg_last_xact_replay_timestamp() has not moved since the promotion.
  • The node does not appear in any surviving node’s pg_stat_replication, so nothing upstream is sending to it.
  • Its log contains repeated could not connect to the primary server entries naming an address that no longer serves that role, or terminating walreceiver due to timeout.
  • Read latency on that node is excellent, because it is serving from a static snapshot with a warm cache. This is why the problem often surfaces as a data complaint rather than an infrastructure alert.

The contrast with a first-level standby is instructive. A direct standby of the old primary fails loudly — its walreceiver cannot connect and lag climbs immediately on the dashboard. A third-level leaf fails silently, because the intermediate node it streams from is still up and simply has nothing new to forward.

Root Cause Analysis

A cascading standby’s timeline comes from its sender, not from the topology as a whole. A node replaying WAL from an intermediate standby learns about a new timeline only if its sender switches to that timeline. When a node one level up is repointed to the new primary and starts replaying timeline 7, it forwards timeline 7 downstream and the leaf follows along. When that node is not repointed, it has nothing to forward, and the leaf waits indefinitely without erroring.

The promoted node’s own children are the exception. If the promoted standby had cascading children, those children were already streaming from it. When it promotes, it begins writing timeline 7 and its existing replication connections simply carry the new timeline forward. Those nodes need no intervention at all — a fact worth knowing, because a script that blindly repoints everything will restart perfectly healthy nodes for no reason.

Divergence propagates downward but not upward. A leaf cannot be ahead of its parent, because everything it holds came through that parent. But when a parent is rewound to the divergence point, its children may now hold WAL the rewound parent no longer has, which makes them diverged relative to their own upstream even though nothing about them changed. This is the case that surprises people: rewinding a middle node can convert three healthy leaves into three diverged ones.

Repointing order for a three-level cascade after promotion Before promotion, the primary feeds two first-level standbys, and one of those feeds two second-level leaves. After the first-level standby B is promoted, its own two children continue streaming and need no action. The sibling standby A must be repointed to B, and only once A is streaming may A's own children be repointed. The numbered order shows level one first, then level two, working outward from the new primary. before primary (dead) A B B1 B2 A1 after promoting B — repoint in this order B — primary timeline 7 B1 — no action B2 — no action A 1 A1 2 B's children inherit timeline 7 through their existing connection — restarting them is pure churn. A1 must wait for A: repoint it first and it attaches to a node still on the dead timeline. Breadth-first, outward from the new primary. Depth-first moves half the tree twice.

Step-by-Step Resolution

Step 1 — Capture the tree from the running system

Documentation goes stale; the running configuration does not.

bash
# On every node, in parallel — what does each one think its upstream is?
for h in pg-a pg-b pg-a1 pg-b1 pg-b2; do
  echo -n "$h: "
  psql -h $h -Atc "SELECT coalesce(setting,'(none)')
                     FROM pg_settings WHERE name='primary_conninfo'"
done

Write the resulting parent-child map down before changing anything. During a failover this map is the only reliable record of what the topology was, and it stops existing the moment you start editing configuration.

Inline verification: every node except the new primary reports a non-empty primary_conninfo, and the set of addresses named forms a connected tree.

Step 2 — Identify the promoted node’s existing children

sql
-- On the NEW primary: who is already streaming from it?
SELECT application_name, client_addr, state
  FROM pg_stat_replication;

Anything listed here is already correct and must be left alone. These are the nodes that were cascading from the standby before it was promoted, and their replication connection carried them onto the new timeline without interruption.

Inline verification: each listed node’s pg_last_xact_replay_timestamp() is advancing.

Step 3 — Repoint level one

For every node whose upstream was the old primary:

bash
# PostgreSQL 12+: primary_conninfo is reloadable, no restart needed.
psql -h pg-a -c "ALTER SYSTEM SET primary_conninfo =
  'host=pg-b port=5432 user=replicator application_name=pg-a';"
psql -h pg-a -c "SELECT pg_reload_conf();"

# Restart the walreceiver so it picks up the new conninfo immediately
# rather than at its next reconnect attempt.
psql -h pg-a -c "SELECT pg_terminate_backend(pid)
                   FROM pg_stat_activity WHERE backend_type='walreceiver';"

pg_reload_conf() alone updates the setting but the existing walreceiver keeps its old connection until it drops. Terminating it forces an immediate reconnect using the new value — the difference between converging in seconds and converging whenever the TCP session happens to time out.

Inline verification: on the new primary, SELECT application_name, state FROM pg_stat_replication; now includes pg-a with state = streaming.

Step 4 — Repoint level two, and only now

bash
# Only after pg-a is confirmed streaming. Its children attach to it.
psql -h pg-a1 -c "ALTER SYSTEM SET primary_conninfo =
  'host=pg-a port=5432 user=replicator application_name=pg-a1';"
psql -h pg-a1 -c "SELECT pg_reload_conf();"
psql -h pg-a1 -c "SELECT pg_terminate_backend(pid)
                    FROM pg_stat_activity WHERE backend_type='walreceiver';"

If a node refuses to attach and its log reports requested starting point ... is ahead of the WAL flush position or new timeline forked off current database system timeline before current recovery point, it has diverged and needs a rewind before it can follow.

Inline verification: on pg-a, pg_stat_replication lists pg-a1 as streaming — a cascading standby forwards WAL and therefore has its own replication view.

Step 5 — Rewind whatever refused

bash
pg_ctl -D /var/lib/postgresql/16/main stop -m fast
pg_rewind --target-pgdata=/var/lib/postgresql/16/main \
          --source-server="host=pg-b user=replicator dbname=postgres" \
          --progress
# pg_rewind clears primary_conninfo — set it again before starting.
psql -h pg-b -c "SELECT 1"   # confirm source reachable first
pg_ctl -D /var/lib/postgresql/16/main start

Rewind against the new primary, not against the node’s intended parent. pg_rewind needs a source that holds the full history back to the divergence point, and an intermediate standby may itself have been rewound and no longer have it.

Inline verification: after restart, the node appears in its intended parent’s pg_stat_replication and its replay timestamp advances.

Configuration Snippet

ini
# postgresql.conf on every cascading node — the settings that make a
# multi-level tree survivable.

# A cascading standby must be able to serve WAL onward to its own children.
# Without this it can replay but not forward, so the level below it starves.
hot_standby = on
max_wal_senders = 10

# Retain enough WAL for a child that briefly disconnects during the repoint.
# Without a slot or wal_keep_size, a child that reconnects after the window
# gets 'requested WAL segment has already been removed' and needs a rebuild.
wal_keep_size = '4GB'

# Prerequisite for pg_rewind. Cannot be enabled without a restart, so it must
# already be on before the failover — verify with SHOW wal_log_hints.
wal_log_hints = on

# Name every node so pg_stat_replication is readable during an incident.
# An unnamed walreceiver shows as 'walreceiver' on every row.
primary_conninfo = 'host=pg-b port=5432 user=replicator application_name=pg-a1'

The wal_keep_size setting deserves attention in cascading topologies specifically. During a repoint every node below the one being moved is disconnected for a few seconds. If the new parent does not retain enough WAL to cover that gap, the child cannot resume and needs a full rebuild — turning a thirty-second repoint into an hours-long base backup. Replication slots are the more robust answer, at the cost of needing slot hygiene so a permanently dead child cannot fill the parent’s disk.

Breadth-first versus depth-first repointing, by work done Two sequences of operations on a five node tree. The breadth-first sequence performs four repoint operations, one per node, and every node is streaming after the fourth. The depth-first sequence performs seven operations because three leaves were attached to parents that had not yet been repointed and had to be moved again, and each redundant move also costs a walreceiver restart and a WAL retention window. breadth-first — 4 operations level 1: A → B level 1: C → B level 2: A1 → A level 2: C1 → C tree converged depth-first — 7 operations, 3 of them wasted A → B A1 → A C1 → C C still on old TL C → B C1 → C again second restart …and two more Each wasted move is another walreceiver restart and another gap the parent's WAL retention must cover. On a tree deep enough, depth-first repointing exhausts wal_keep_size and turns repoints into rebuilds.

Verification and Rollback

Walk the tree outward and confirm every edge:

bash
# A tree is healthy when every non-primary node appears in exactly one
# other node's pg_stat_replication, and every replay timestamp is moving.
for h in pg-b pg-a pg-c; do
  echo "== $h is sending to:"
  psql -h $h -Atc "SELECT application_name || ' ' || state
                     FROM pg_stat_replication ORDER BY application_name"
done

# And from the receiving side — the definitive check for a silent orphan.
for h in pg-a pg-c pg-a1 pg-c1; do
  echo -n "$h replay age: "
  psql -h $h -Atc "SELECT now() - pg_last_xact_replay_timestamp()"
done

A node whose replay age keeps growing while its parent does not list it is the orphan case. A node whose replay age is small and constant has caught up and is idle, which is normal on a quiet cluster — distinguish the two by writing a marker to the primary and watching it arrive.

sql
-- On the new primary: a marker that must reach every leaf.
CREATE TABLE IF NOT EXISTS topology_probe(id serial, at timestamptz default now());
INSERT INTO topology_probe DEFAULT VALUES;
bash
# On every leaf — the marker is the end-to-end proof the whole path works.
for h in pg-a1 pg-c1 pg-b1 pg-b2; do
  echo -n "$h: "; psql -h $h -Atc "SELECT max(at) FROM topology_probe"
done

Rollback. There is no meaningful rollback for a repoint — the old primary is gone and the old topology no longer exists. What you can do is undo a wrong repoint: point the node back at its correct parent and restart the walreceiver. If a node was rebuilt unnecessarily, nothing is lost except time. The genuine risk to guard against is repointing a node at an upstream that is itself diverged, which propagates the problem downward; always confirm the intended parent is streaming from the new primary before attaching anything to it.

Edge Cases and Gotchas

A synchronous standby cannot be a cascade leaf

synchronous_standby_names matches on application_name of nodes with a direct replication connection to the primary. A cascading node connects to an intermediate standby, so the primary never sees it and it can never satisfy a synchronous requirement. If your topology has only one first-level node and it is the promotion candidate, the promotion leaves you with no synchronous standby at all until a second first-level node exists.

Replication slots do not cascade automatically

A slot on the primary protects WAL for a direct standby. It does not protect WAL for that standby’s children — those need their own slots on the intermediate node. After a promotion, slots on the old primary are gone and slots on intermediate nodes may reference children that have since been repointed elsewhere. Audit them, because an orphaned slot pins WAL forever:

sql
SELECT slot_name, active, pg_size_pretty(
         pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
  FROM pg_replication_slots WHERE NOT active ORDER BY retained DESC;

A repoint during heavy write load can outrun WAL retention

The disconnect window during a repoint is short, but on a high write-rate cluster a few seconds can be gigabytes of WAL. If the new parent’s wal_keep_size is smaller than what accumulates during the gap, the child cannot resume. Either raise retention before starting the repoint, or use slots, or perform the repoint during a write-quiet window — the same consideration that governs cross-AZ replica setup.

FAQ

Do cascading replicas automatically follow a promotion?

Only the ones whose upstream is the promoted node itself. Those pick up the new timeline automatically because their sending server is the node that created it. Every other node in the tree is still streaming from an upstream that either no longer exists or is now on a dead timeline, and each must be repointed explicitly.

Should I repoint the tree breadth-first or depth-first?

Breadth-first, working outward from the new primary. A depth-first walk repoints a leaf at a parent that has not itself been repointed yet, so the leaf attaches to a node still on the dead timeline and has to be moved twice. Breadth-first guarantees that when you move a node, its new upstream is already correct.

Can a cascade leaf diverge when its parent did not?

No — a leaf can only have received what its parent sent, so it can never be ahead of its parent. But the reverse matters: a parent that diverged and was rewound has effectively rewound its children’s future too, and those children may now be ahead of their rewound parent. Test each node against the new primary’s switchpoint rather than assuming the tree is uniform.

Is a cascading topology worth the extra failover complexity?

It is when the alternative is many standbys all streaming directly from the primary across an expensive link — the cascade sends one copy over the wide-area link and fans it out locally, which is often the whole reason for a multi-region design. The cost is exactly this: failover becomes an ordered tree walk instead of a flat repoint, and it must be scripted rather than improvised.

← Back to Replica Failover & Promotion Mechanics