When Adding Read Replicas Stops Improving Throughput

Problem statement: the second replica roughly doubled read capacity, the third added noticeably less, and the fourth appears to have added nothing at all.

Read replicas scale one thing linearly and several things not at all. Understanding which of the non-scaling costs you have hit determines whether the remedy is a bigger node, a different workload split, or a fundamentally different architecture. This page identifies the four ceilings and the measurement that distinguishes them. It extends read scaling trade-offs in high-traffic applications.

Symptom Identification

• Aggregate read throughput at the same latency has risen by much less than one node’s worth after the most recent addition.
• Replication lag across the whole fleet rose when the new replica was added, not just on the new node.
• The primary’s CPU did not fall despite the extra read capacity.
• The new replica’s buffer cache hit ratio is markedly worse than its siblings’.
• The new replica is receiving little traffic while the others are unchanged.

The last symptom is the one to rule out first, because it is not a scaling limit at all — it is a routing problem, and the causes are in why DNS round robin unbalances read replica traffic. Diagnosing a ceiling when the node is simply idle wastes a great deal of time.

Root Cause Analysis

Four costs do not divide across replicas, and each produces a different ceiling.

1 — Replay is per node and largely serial. Every replica applies the entire WAL stream. Ten replicas each apply 100% of the writes; the work is duplicated, not shared. On PostgreSQL, replay is essentially single-threaded, so each replica’s ability to keep up is bounded by one core and its storage. As the primary’s write rate approaches that bound, every replica lags, and a new one faces the same ceiling on arrival.

2 — The primary pays per replica. Each streaming replica consumes a wal_sender process, a slice of network egress equal to the full WAL volume, and — with hot_standby_feedback on — additional bloat because vacuum is held back for the slowest replica. Adding replicas therefore increases load on the node you were trying to relieve.

3 — Memory is per node and does not pool. Two replicas with 32 GB each do not give you 64 GB of cache for one working set; they give you two nodes each caching the same hot pages. If the working set exceeds one node’s memory, every replica is disk-bound, and adding another adds another disk-bound node.

4 — Connection and coordination costs grow. More nodes means more health checks, more lag probes, more pool slots reserved per node, and more time in the freshness gate rejecting candidates during a lag event.

The consequence is a curve that flattens. Read capacity rises roughly linearly with node count while replay burden stays constant per node and primary cost rises linearly — so the marginal benefit falls, and beyond some point the added primary cost exceeds the added read capacity’s value.

Step-by-Step Resolution

Step 1 — Rule out the idle-node case first

-- On each replica. A node with near-zero traffic is a routing problem.
SELECT current_setting('cluster_name', true) AS node,
       (SELECT count(*) FROM pg_stat_activity
         WHERE backend_type='client backend')       AS conns,
       (SELECT sum(xact_commit) FROM pg_stat_database
         WHERE datname=current_database())          AS commits;

Inline verification: every replica shows a comparable share of connections and commits. If one does not, stop here — the problem is distribution, not capacity.

Step 2 — Measure replay headroom

-- On each replica: how much WAL is waiting to be applied, and is the gap growing?
SELECT pg_wal_lsn_diff(pg_last_wal_receive_lsn(),
                       pg_last_wal_replay_lsn())    AS replay_backlog_bytes,
       now() - pg_last_xact_replay_timestamp()      AS replay_age;

# Is the replay process CPU-bound? On a standby this is the startup process.
top -b -n1 -o %CPU | grep -E 'startup|recovering' | head -3

A replay backlog that grows under normal write load, with the startup process near 100% of one core, is the replay ceiling. No number of replicas fixes it.

Inline verification: during peak writes, replay_backlog_bytes is stable rather than trending upward on every node.

Step 3 — Check what the primary pays per replica

-- On the primary: one row per streaming replica, each costing a sender
-- process and full WAL egress.
SELECT application_name, state,
       pg_size_pretty(pg_wal_lsn_diff(sent_lsn, replay_lsn)) AS behind,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)) AS unsent
  FROM pg_stat_replication ORDER BY application_name;

-- With hot_standby_feedback on, the SLOWEST replica holds back vacuum
-- everywhere, so one lagging node bloats the primary for all of them.
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
  FROM pg_replication_slots ORDER BY retained DESC;

Inline verification: total WAL egress is roughly the primary’s WAL generation rate multiplied by replica count — confirm the network path can carry it.

Step 4 — Test whether the workload is cache-resident

-- Buffer cache hit ratio per replica. Below about 0.98 on an OLTP workload
-- means the node is going to disk regularly and is not delivering the
-- throughput its CPU count suggests.
SELECT sum(blks_hit)::numeric / nullif(sum(blks_hit + blks_read), 0) AS hit_ratio
  FROM pg_stat_database WHERE datname = current_database();

-- How big is the hot set relative to shared_buffers?
SELECT pg_size_pretty(sum(pg_relation_size(relid))) AS heap_size
  FROM pg_stat_user_tables WHERE seq_scan + idx_scan > 0;

Inline verification: if hit ratio is low and the hot set exceeds shared_buffers plus OS cache, a larger replica will out-perform an additional one — test with a single resized node before buying several.

Step 5 — Apply the remedy the measurement implies

Inline verification: the chosen remedy addresses the measurement that was actually binding, not the one that was easiest to change.

Configuration Snippet

# postgresql.conf on the PRIMARY — settings that change how much each
# replica costs and how fast replay can go.

# WAL volume is the input to every per-replica cost. Compression trades
# primary CPU for less egress per replica — worth it above ~3 replicas.
wal_compression = zstd

# Full-page writes after a checkpoint dominate WAL volume. Longer intervals
# mean less WAL and therefore less replay work on EVERY replica.
checkpoint_timeout = '15min'
max_wal_size = '16GB'

# One sender per replica, plus headroom for base backups.
max_wal_senders = 12

# hot_standby_feedback = on (set on the REPLICA) holds back vacuum on the
# primary for the slowest replica. Great for query stability, bad for bloat
# once you have many replicas — reconsider it as the fleet grows.

# postgresql.conf on each REPLICA — settings that raise the replay ceiling.

# Prefetch pages that replay is about to need. On PostgreSQL 15+ this is
# the single most effective setting for a replay-bound standby on
# network-attached storage, where each miss costs a round trip.
recovery_prefetch = try
wal_decode_buffer_size = '2MB'

# Give replay room before it is forced to wait on a conflicting query.
max_standby_streaming_delay = '30s'

# Cache sizing: the hot set must fit here or the node is disk-bound
# regardless of how many siblings it has.
shared_buffers = '16GB'
effective_cache_size = '48GB'

wal_compression and checkpoint_timeout deserve attention specifically because their benefit scales with replica count. Halving WAL volume halves egress on the primary, halves network transfer for each replica, and halves the replay work every replica must perform — one setting improving the constraint on every node simultaneously. On a fleet of six replicas that is a far larger effect than on a fleet of one, and it is often the cheapest way to move the flattening point to the right.

Verification and Rollback

Measure marginal capacity properly — the common error is comparing peak throughput on different days:

# Ramp against a fixed latency objective, before and after, same conditions.
# The number that matters is sustained tps at your p99 ceiling, not peak tps.
pgbench -h read-endpoint -c 64 -j 16 -T 300 -S -P 30 app | tail -6

Run against the read endpoint (the balancer), not against a single node, so the measurement includes the routing tier. Repeat at the same time of day with the same background load.

Confirm where the primary’s load went:

-- If primary CPU did not fall after adding replica capacity, reads never
-- left it. This is a routing problem masquerading as a scaling limit.
SELECT round(100 * sum(total_exec_time) FILTER (WHERE query ~* '^\s*select')
             / nullif(sum(total_exec_time), 0)) AS pct_time_in_selects
  FROM pg_stat_statements;

Rollback. Removing a replica is straightforward and worth doing when the measurement says it is not earning its cost:

-- On the primary, after removing the node from the read pool and draining it.
SELECT pg_drop_replication_slot('replica_4_slot');

Drain it from the balancer first and confirm the remaining nodes have headroom, using the same n/(n−1) check as in draining a read replica for maintenance without dropping queries. Dropping the slot is the irreversible step: re-adding the node later requires a fresh base backup, so do it only once you are confident the node is genuinely surplus.

Edge Cases and Gotchas

hot_standby_feedback makes the slowest replica everyone’s problem

With it enabled, the primary holds back vacuum to preserve row versions the slowest standby query still needs. Add replicas and the probability that some node has a long-running query at any moment rises, so vacuum is held back more often and the primary bloats. On a large fleet, consider disabling it on the replicas running long analytical queries and accepting cancellations there instead — the trade-off is examined in detecting hot standby query cancellations on Postgres replicas.

Cascading replication changes the primary’s cost but not the replay ceiling

Streaming from an intermediate standby removes senders and egress from the primary, which addresses ceiling 2 — but every leaf still applies the complete WAL stream, so ceiling 1 is unchanged. Cascading is a bandwidth and primary-load optimisation, not a replay-capacity one, and it is often proposed for the wrong ceiling.

Aurora and similar shared-storage engines have a different curve

Where replicas read from shared storage rather than applying WAL locally, ceiling 1 largely does not apply — there is no per-node replay. The curve still flattens, for different reasons (shared storage throughput, the writer’s own limits), and the flattening point is usually much further right. Do not carry conclusions from a streaming-replication fleet to a shared-storage one; the comparison is in Aurora replicas vs RDS read replicas for read scaling.

FAQ

Related

← Back to Read Scaling Trade-offs in High-Traffic Applications

• Aurora Replicas vs RDS Read Replicas for Read Scaling — an architecture where the replay ceiling largely does not apply.
• Redis Cache Versus Read Replica for Hot Read Paths — the alternative when the constraint is round trips rather than capacity.
• Load Balancing Reads Across a Replica Pool — making sure the capacity you already have is being used.