Why DNS Round Robin Unbalances Read Replica Traffic

Problem statement: three replicas share one DNS name with round-robin records, and traffic is split roughly 60/30/10 instead of evenly — with no configuration anywhere that asked for that.

DNS round robin is the cheapest thing that looks like load balancing, and for a database read tier it is close to a no-op. This page explains precisely why, shows how to confirm that DNS rather than query cost is the culprit, and covers what to replace it with. The broader treatment of replica selection is in load balancing reads across a replica pool.

Symptom Identification

The signature of DNS-caused skew, as distinct from other imbalances:

  • Connection counts are uneven, not just query counts. SELECT count(*) FROM pg_stat_activity returns very different numbers per replica.
  • The skew is stable. It does not drift over hours or vary with traffic; the same replica has been the busy one since the last deploy.
  • It changes only at deploys or restarts — and then jumps to a new, equally arbitrary, equally stable split.
  • A replica that was recently restarted has near-zero connections and stays that way indefinitely.
  • Per-connection query rates are similar, so the imbalance is entirely explained by how many connections each replica holds.

Contrast this with query-cost skew, where connection counts are even and CPU is not, which points at expensive digests rather than at DNS. The two have completely different remedies, so distinguish them before acting — the method is in weighted vs least-connections balancing.

Root Cause Analysis

DNS round robin balances resolutions, and a database client barely resolves. Rotating records works when a large population of independent clients each performs a fresh lookup — that is the web-browser case it was designed for. A read tier is the opposite: perhaps forty application processes, each resolving once at start-up, each then holding a pool of connections for the process lifetime.

Forty samples from a three-way rotation do not produce an even split, and whatever split they do produce is then frozen for as long as the processes live. That is the entire mechanism.

Four layers compound it, and each one alone would be enough.

Resolver caching. The client’s stub resolver, the container’s DNS cache, and any upstream forwarder each cache the answer. The rotation happens at the authoritative server; downstream caches serve one fixed ordering to everyone who asks during the TTL.

Client-side address selection. Given multiple A records, most client libraries take the first. Some sort them — glibc’s getaddrinfo applies RFC 3484/6724 rules that can order addresses deterministically by prefix similarity, which on a flat internal network makes every client in a subnet prefer the same replica.

Connection pooling. Once a connection is established, DNS is irrelevant. A pool holding twenty connections opened at start-up will keep using those twenty sockets for as long as the process runs.

Absence of a redistribution trigger. Nothing ever forces re-resolution. A replica that returns from maintenance is in the DNS record but receives nothing, because no client is opening connections.

Where the DNS rotation is lost between the record and the query A vertical path from the authoritative DNS server down to an executed query. The authoritative server rotates the three records evenly. The caching resolver stores one ordering and serves it unchanged for the whole time to live, so every client in that period sees the same order. The client library takes the first address, or sorts addresses deterministically by prefix. The connection pool then holds the resulting sockets for hours. By the time a query executes, four layers have each removed variation, and the effective distribution is fixed. authoritative DNS rotates r1, r2, r3 evenly ✓ the only layer that balances caching resolver stores ONE order for the whole TTL rotation collapses to a constant client library takes the first, or sorts by prefix a whole subnet can prefer one node connection pool holds the socket for hours — never re-resolves freezes whatever the layers above produced the decisive layer: fix this one first

Step-by-Step Resolution

Step 1 — Confirm the skew is in connections, not in cost

sql
-- Run against each replica. Compare BOTH numbers across nodes.
SELECT (SELECT count(*) FROM pg_stat_activity
         WHERE backend_type = 'client backend')            AS connections,
       (SELECT sum(xact_commit) FROM pg_stat_database
         WHERE datname = current_database())                AS commits;

If connections differ widely and commits-per-connection are similar, the cause is address selection. If connections are similar and commits differ, the cause is elsewhere.

Inline verification: the ratio of max to min connection count across replicas exceeds about 1.5 while commits-per-connection are within 20% of each other.

Step 2 — Trace where the pinning happens

bash
# What the authoritative server returns — should rotate between calls.
for i in 1 2 3; do dig +short @10.0.0.2 pg-read.internal | tr '\n' ' '; echo; done

# What the client's resolver returns — often does NOT rotate.
for i in 1 2 3; do getent ahosts pg-read.internal | awk '{print $1}' \
   | tr '\n' ' '; echo; done

# What a running process actually connected to.
ss -tnp state established '( dport = :5432 )' | awk '{print $4, $5}' | sort | uniq -c

The three outputs usually disagree, and the disagreement localises the problem: authoritative rotates, resolver does not, and the process is pinned to whatever the resolver said the day it started.

Inline verification: the third command shows established connections concentrated on one or two addresses.

Step 3 — Apply the interim mitigation: bound connection lifetime

This does not replace DNS round robin, but it converts a permanently frozen distribution into one that continuously re-samples.

yaml
# HikariCP (JVM)
maxLifetime: 300000          # 5 min — connections age out and re-resolve
keepaliveTime: 30000

# PgBouncer, if it sits between the app and the replicas
server_lifetime = 300
server_idle_timeout = 60
python
# SQLAlchemy
engine = create_engine(
    "postgresql+psycopg://app@pg-read.internal/app",
    pool_recycle=300,        # seconds; recycled connections re-resolve
    pool_pre_ping=True,
)

Inline verification: over a 15-minute window, per-replica connection counts converge rather than staying fixed — watch pg_stat_activity counts every minute.

Step 4 — Move selection off DNS

Two workable replacements, in increasing order of capability.

Multi-host connection string. Supported by libpq (and therefore psql, psycopg, and many others), by the PostgreSQL JDBC driver, and by several MySQL drivers:

text
postgresql://app@r1.internal:5432,r2.internal:5432,r3.internal:5432/app
  ?target_session_attrs=prefer-standby
  &load_balance_hosts=random

load_balance_hosts=random is the important parameter: without it libpq tries hosts strictly in order, so every client connects to r1 until it is unavailable. target_session_attrs=prefer-standby additionally keeps reads off the primary if it is in the list.

A real balancer. A per-zone HAProxy or ProxySQL in front of the replicas gives health awareness, weights, slow-start, and lag-aware selection — everything DNS structurally cannot provide.

Inline verification: with the multi-host string, restart one application instance repeatedly and confirm it lands on different replicas across restarts.

Step 5 — Verify the distribution holds across disruptions

promql
# Share per replica; should stay near even through a deploy and a restart.
sum by (instance) (pg_stat_activity_count{role="replica", state="active"})
  / ignoring(instance) group_left
    sum (pg_stat_activity_count{role="replica", state="active"})

The real test is not the steady state but the recovery: restart one replica and confirm its share returns to roughly a third within a connection lifetime.

Inline verification: after a replica restart, its traffic share recovers to within a few percent of its siblings within maxLifetime.

Configuration Snippet

text
# What to replace the DNS name with — a per-zone proxy that makes a real
# decision per connection, with the health signals DNS cannot express.
listen pg_read
    bind 0.0.0.0:6432
    mode tcp
    balance leastconn
    option pgsql-check user haproxy_check
    default-server inter 2s fall 3 rise 5 slowstart 60s maxconn 120
    server r1 10.0.1.11:5432 check
    server r2 10.0.1.12:5432 check
    server r3 10.0.1.13:5432 check
ini
# If a proxy is genuinely not an option, the multi-host string is the next
# best thing. These three parameters matter more than the host list itself.
[database]
url = postgresql://app@r1.internal,r2.internal,r3.internal/app
# random: without this, every client tries r1 first and pins there.
load_balance_hosts = random
# prefer-standby: keeps reads off the primary if it appears in the list.
target_session_attrs = prefer-standby
# Short connect timeout so an unreachable host is skipped quickly rather
# than blocking the whole connection attempt.
connect_timeout = 3

Keep the DNS name in place either way, pointing at the proxy or as an alias — the name is a useful indirection, and the problem was never the name. What has to change is that something along the path makes a decision rather than passively accepting a cached ordering.

Traffic share over a week: DNS round robin versus a proxy Two line charts spanning a week. Under DNS round robin the three replicas hold a stable but uneven split of roughly sixty, thirty and ten percent, with abrupt step changes at each deploy that produce a new equally arbitrary split, and one replica flatlining near zero after its restart on day four. Under a proxy the three lines sit together near thirty three percent throughout, dip briefly when one replica restarts on day four, and return to even within a minute. DNS round robin deploy r3 restart stable, skewed, and reshuffled only by deploys r3 never recovers after its restart proxy with leastconn + slowstart r3 restart even throughout; r3 ramps back within a minute Both charts show the same fleet and the same traffic. The only difference is whether anything makes a decision. Solid, dashed and dotted are r1, r2 and r3 respectively in both charts.

Verification and Rollback

Confirm the fix from the database’s point of view:

sql
-- Connection distribution by source. Should be even across replicas and
-- should include every application host, not a subset.
SELECT client_addr, count(*) FROM pg_stat_activity
 WHERE backend_type = 'client backend'
 GROUP BY client_addr ORDER BY 2 DESC LIMIT 20;

Confirm it survives a restart, which is the case DNS round robin fails:

bash
# Restart one replica, then watch its share recover.
systemctl restart postgresql@16-main    # on r3
watch -n5 'for h in r1 r2 r3; do echo -n "$h "; \
  psql -h $h -Atc "SELECT count(*) FROM pg_stat_activity WHERE backend_type='"'"'client backend'"'"'"; done'

Under a proxy with slow-start, r3 climbs back to parity within a minute. Under bounded-lifetime DNS, within one maxLifetime. Under unbounded DNS, never.

Rollback. The interim mitigation (bounded connection lifetime) is safe to keep permanently and rarely needs reverting — its only cost is a low rate of connection establishment, which matters solely if authentication is expensive. If you moved to a multi-host connection string and need to revert, restoring the single DNS name is a configuration change and a rolling restart. If you introduced a proxy, revert by pointing the connection string back at the DNS name; note that this re-introduces the skew immediately for any process that restarts afterwards, and gradually for the rest.

Edge Cases and Gotchas

getaddrinfo sorting can defeat rotation entirely

RFC 6724 destination address selection sorts candidate addresses by, among other things, longest matching prefix with the source address. On a flat internal network where replicas share a subnet with the clients, this can produce a deterministic ordering that is identical for every client in a given subnet — so the authoritative rotation is not merely cached, it is actively re-sorted away. This is why dig showing a rotation and getent ahosts showing a fixed order is such a common pairing.

Kubernetes headless services have the same problem with extra caching

A headless service returns all pod IPs, and CoreDNS caches the answer. Client-side caching in the pod’s resolver adds another layer, and a client library that resolves once at start-up completes the picture. Headless services are useful for discovery; they are not a load balancer, and treating them as one produces exactly the skew described here.

Short TTLs cost queries and fix nothing

Dropping the TTL to a few seconds increases resolver traffic and can add latency to connection establishment, while doing nothing about the pooled connections that carry all the traffic. If you find yourself tuning TTL to fix replica balance, the mechanism you actually need is connection lifetime — DNS is not on the path of an established connection at all.

FAQ

Does DNS round robin balance anything at all?

It rotates the order of records in the answer, which balances resolutions across a large population of independent, short-lived clients. A database read tier is the opposite case: a small number of long-lived processes each resolving once and then holding pooled connections for hours. In that population the rotation is sampled a handful of times, so the resulting distribution is essentially random and stays fixed.

Why does one replica get almost no traffic after a restart?

Because nothing triggers re-resolution. While that replica was down, every client that reconnected chose one of the survivors and has held those connections since. When the replica returns, the DNS record includes it again but no client is establishing new connections, so it stays empty. Only connection turnover redistributes traffic, and with unbounded connection lifetime there is none.

Do multi-host connection strings fix this?

Partly, and they are a genuine improvement. libpq and several drivers accept a comma-separated host list and can select randomly rather than in order, which spreads new connections across replicas without a proxy. What they do not give you is health awareness beyond connectivity, lag awareness, or the ability to move an existing connection — so they suit simple deployments and do not replace a balancer in demanding ones.

Is a short TTL enough to fix the skew?

No. A short TTL controls how often a client re-resolves, but a client with an established pooled connection does not care what DNS says — it keeps using the socket it already has. The TTL only affects clients opening new connections, so without bounded connection lifetime a five-second TTL and a five-day one produce the same distribution.

← Back to Load Balancing Reads Across a Replica Pool