Zone-Aware Read Routing to Cut Cross-AZ Data Transfer

Problem statement: your application instances and read replicas are spread across three availability zones, every read has a two-in-three chance of crossing a zone boundary, and you are paying for that in both latency and per-gigabyte transfer charges.

Keeping reads in-zone is a straightforward optimisation with one serious trap: done as a hard partition it converts each zone into a single point of failure for the callers in it. This page implements it as a preference with automatic spill-over, which captures the benefit without the trap. It sits within load balancing reads across a replica pool.

Symptom Identification

Zone-aware routing is worth considering when:

  • Cross-AZ data-transfer charges are a visible line item, and the traffic breakdown attributes most of it to database ports rather than to inter-service calls.
  • p50 query latency measured from the application is meaningfully higher than the same query’s execution time measured on the replica — the gap is network round trip.
  • A request that issues many sequential small reads has a latency profile dominated by round trips rather than by execution.
  • Your application tier and replica tier are both spread across the same zones, so an in-zone target usually exists.

It is not worth doing when the read path issues few, large queries — there the transfer cost is the same wherever the query runs and the round-trip saving is amortised over a big result — or when your replicas are concentrated in fewer zones than your application instances, in which case most reads must cross anyway.

Root Cause Analysis

Cross-zone traffic is billed and it is slow, in that order of importance. The latency difference between an in-zone and cross-zone hop within one region is small in absolute terms — typically well under a millisecond — but it is paid per round trip. A read path that issues thirty sequential queries pays it thirty times, which is why the benefit is so much larger for chatty ORM code than for a service issuing one aggregate.

The transfer charge is the more predictable win. Read result sets are usually the highest-volume database traffic a service produces, and cross-zone transfer is charged per gigabyte in each direction on the major providers. The calculation is simple and worth doing before any configuration changes.

The obvious implementation is the dangerous one. Configuring each application zone to use only its local replicas produces perfect locality and a per-zone failure domain: when zone B’s replicas are down, or lagging past the freshness gate, every application instance in zone B loses its read path entirely — even though healthy replicas exist a millisecond away in zones A and C.

The fix is a preference with a fallback tier. HAProxy’s backup keyword expresses exactly this: backup servers receive no traffic while any non-backup server is available, and take over automatically when none is. The result is in-zone traffic under normal conditions and cross-zone traffic only when the local option has genuinely gone.

The capacity precondition is easy to skip. Zone preference concentrates each zone’s read load onto that zone’s replicas. If zone A hosts half your application instances but only a third of your replicas, enabling zone preference moves zone A’s replicas from comfortable to saturated on the day you turn it on. Check the ratio first.

Cross-zone read traffic before and after zone preference On the left, application instances in three zones send reads to all replicas uniformly, so two of every three reads cross a zone boundary and incur transfer charges and an extra round trip. On the right, each zone's applications prefer their local replicas, so almost all reads stay in-zone, while dashed lines show that out-of-zone replicas remain configured as backup servers and take traffic only when the local ones are unavailable. uniform balancing — 2 of 3 reads cross a zone app · a app · b app · c r · a r · b r · c 6 of 9 paths are billed cross-zone hops zone preference with backup tier app · a app · b app · c r · a r · b r · c solid = normal path · faint dashed = backup, idle until needed The backup paths cost nothing while unused and are the entire difference between a preference and a partition.

Step-by-Step Resolution

Step 1 — Quantify what crossing actually costs you

sql
-- Bytes returned by read queries, as a proxy for egress volume.
-- Run on each replica; the sum across replicas is your monthly read transfer.
SELECT sum(pg_column_size(t)) FROM (SELECT 1) t;  -- placeholder for shape

-- Practical version: pg_stat_statements totals, scaled.
SELECT round(sum(rows * 200) / 1024.0^3, 1) AS approx_gb_returned_est
  FROM pg_stat_statements WHERE query ~* '^\s*select';
-- 200 bytes/row is a rough average; replace with your measured row width.
bash
# The authoritative number comes from the cloud provider, not the database.
aws ce get-cost-and-usage \
  --time-period Start=2026-07-01,End=2026-07-31 --granularity MONTHLY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"USAGE_TYPE_GROUP","Values":["EC2: Data Transfer - Inter AZ"]}}'

Inline verification: you have a monthly figure in currency. If it is small relative to the engineering effort, stop here — this optimisation is not free to operate.

Step 2 — Check per-zone capacity before concentrating load

promql
# Application instances per zone — how the read load will distribute.
count by (zone) (up{job="app"})

# Replica capacity per zone — what will have to absorb it.
sum by (zone) (machine_cpu_cores{role="replica"})

Compare the two ratios. If zone A holds 45% of application instances and 33% of replica capacity, enabling zone preference raises zone A’s replica load by roughly 35% relative to today. Confirm that headroom exists, or rebalance the replicas first.

Inline verification: for every zone, projected load (its share of app instances × total read rate) is below 60% of that zone’s replica capacity — leaving room for one replica to fail.

Step 3 — Configure preference, not partition

One backend per zone, with the other zones’ replicas present as backup:

text
backend pg_read_zone_a
    balance leastconn
    option pgsql-check user haproxy_check
    default-server inter 2s fall 3 rise 5 slowstart 60s

    # Local — normal traffic goes here.
    server a1 10.0.1.11:5432 check weight 100
    server a2 10.0.1.12:5432 check weight 100

    # Remote — idle while any local server is up; automatic spill-over if not.
    server b1 10.0.2.11:5432 check weight 100 backup
    server c1 10.0.3.11:5432 check weight 100 backup

    # Use ALL backups together rather than only the first one listed,
    # so spill-over spreads across zones instead of hammering one replica.
    option allbackups

option allbackups is easy to miss and matters a great deal. Without it HAProxy sends all spill-over traffic to the first backup server only, so a zone-A outage concentrates zone A’s entire read load onto b1 alone.

Inline verification: echo "show stat" | socat stdio /var/run/haproxy.sock | grep pg_read_zone_a shows zero sessions on b1 and c1 while a1 and a2 are up.

Step 4 — Make the application select the right backend

yaml
# Kubernetes: expose the node's zone to the pod, then pick the backend by it.
env:
  - name: AZ
    valueFrom:
      fieldRef:
        fieldPath: metadata.labels['topology.kubernetes.io/zone']
  - name: DB_READ_ENDPOINT
    value: "pg-read-$(AZ).internal:5432"    # resolves per-zone
python
# Non-Kubernetes: read the zone from instance metadata once at start-up.
# Do NOT call this per request — it is a network round trip.
import os, urllib.request, functools

@functools.lru_cache(maxsize=1)
def availability_zone() -> str:
    if (z := os.environ.get("AZ")):
        return z
    req = urllib.request.Request(
        "http://169.254.169.254/latest/meta-data/placement/availability-zone")
    with urllib.request.urlopen(req, timeout=1) as r:
        return r.read().decode()

READ_ENDPOINT = f"pg-read-{availability_zone()}.internal"

Inline verification: each application instance logs the endpoint it resolved at start-up, and the distribution of endpoints across the fleet matches the zone distribution.

Step 5 — Drill the zone-loss case

bash
# Take zone A's replicas out and confirm spill-over rather than failure.
echo "disable server pg_read_zone_a/a1" | socat stdio /var/run/haproxy.sock
echo "disable server pg_read_zone_a/a2" | socat stdio /var/run/haproxy.sock

# Reads from zone A should continue, now served by b1 and c1.
echo "show stat" | socat stdio /var/run/haproxy.sock | grep pg_read_zone_a

# Restore.
echo "enable server pg_read_zone_a/a1" | socat stdio /var/run/haproxy.sock
echo "enable server pg_read_zone_a/a2" | socat stdio /var/run/haproxy.sock

Inline verification: during the drill, zone A’s read error rate stays at zero and its p99 latency rises by roughly one cross-zone round trip — the expected, acceptable degradation.

Configuration Snippet

text
# Complete per-zone backend set. One backend per zone; each lists its own
# replicas as primary and the others as backups.
global
    log stdout format raw local0

defaults
    mode tcp
    timeout connect 3s
    timeout client 30m
    timeout server 30m
    option tcplog

listen pg_read_a
    bind 0.0.0.0:6432
    balance leastconn
    option pgsql-check user haproxy_check
    option allbackups
    default-server inter 2s fall 3 rise 5 slowstart 60s maxconn 120
    server a1 10.0.1.11:5432 check weight 100
    server a2 10.0.1.12:5432 check weight 100
    server b1 10.0.2.11:5432 check weight 100 backup
    server c1 10.0.3.11:5432 check weight 100 backup

listen pg_read_b
    bind 0.0.0.0:6433
    balance leastconn
    option pgsql-check user haproxy_check
    option allbackups
    default-server inter 2s fall 3 rise 5 slowstart 60s maxconn 120
    server b1 10.0.2.11:5432 check weight 100
    server b2 10.0.2.12:5432 check weight 100
    server a1 10.0.1.11:5432 check weight 100 backup
    server c1 10.0.3.11:5432 check weight 100 backup

Run the proxy itself in each zone rather than centrally. A single central proxy re-introduces the cross-zone hop it was deployed to remove — every read then crosses from the application to the proxy’s zone and possibly back again. Deploying one proxy per zone, alongside the applications it serves, is what makes the local path genuinely local.

The maxconn 120 per server is doing double duty here. Beyond bounding connection use, it means a zone-outage spill-over cannot open unbounded connections against the remaining zones’ replicas — the excess queues at the proxy where it is visible and bounded, instead of exhausting max_connections on a replica that is now serving two zones.

How zone preference redistributes replica load Two grouped bar charts across three zones. Before zone preference, read load is even across all replicas regardless of where the callers are. After enabling it, each zone's replicas carry that zone's share of application instances: zone A rises well above the previous level because it hosts forty-five percent of the application tier, zone C falls. A dashed capacity line shows zone A crossing sixty percent utilisation, indicating capacity must be added there before enabling the change. replica load per zone 60 % capacity — the line to stay under zone A 45 % of app instances over the line zone B 33 % of app instances zone C 22 % of app instances uniform balancing zone preference Add replica capacity in zone A before enabling, not after.

Verification and Rollback

Confirm reads are actually staying local:

sql
-- On each replica: which subnets are its clients in?
SELECT client_addr, count(*) AS conns
  FROM pg_stat_activity
 WHERE backend_type = 'client backend'
 GROUP BY client_addr ORDER BY conns DESC;

Compare the client addresses against your zone subnet allocations. A replica in the 10.0.1.0/24 (zone A) subnet should be serving almost exclusively 10.0.1.0/24 clients under normal conditions.

Confirm the cost actually fell:

bash
# Compare the inter-AZ transfer line item month over month.
aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-08-01 --granularity MONTHLY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"USAGE_TYPE_GROUP","Values":["EC2: Data Transfer - Inter AZ"]}}'

Allow a full billing period. Partial-month comparisons are misleading because transfer volume tracks traffic, which is not uniform across a month.

Rollback. Reverting is a config change with no data implications:

bash
# Immediate: promote the backups to normal servers at runtime so traffic
# spreads across all zones again.
echo "set server pg_read_a/b1 weight 100" | socat stdio /var/run/haproxy.sock

# Permanent: remove the `backup` keyword from the remote server lines and
# reload, or point every application at a single shared backend.
systemctl reload haproxy

Revert if per-zone replica load turns out higher than projected, or if a zone’s replicas prove unable to carry their zone alone. Both symptoms appear as rising p99 latency confined to one zone, which is a distinctive and easily-alertable shape.

Edge Cases and Gotchas

Autoscaling changes the zone distribution under you

The capacity check in step 2 is a snapshot. An autoscaler that adds instances wherever capacity is cheapest will drift the zone distribution over time, and the zone that was at 45% may reach 60% without anyone changing a configuration. Alert on per-zone application instance share, not only on replica utilisation, so the drift is visible before it becomes a capacity event.

A zone’s replicas can be “up” and unusable

Zone preference interacts with the freshness gate: if zone A’s replicas are all healthy but all lagging past the budget, the backup mechanism does not engage, because HAProxy sees the local servers as up. The spill-over you want in that case has to come from the freshness gate’s own fallback, not from the zone configuration. Make sure the two mechanisms are aware of each other — the lag agent’s drain verdict is what makes a lagging local replica look “down” to HAProxy and thereby trigger the backup tier.

Cross-region is not the same problem

The backup pattern works within a region, where the fallback costs a millisecond. Applying the same shape across regions makes the fallback cost tens of milliseconds per query plus cross-region transfer at a much higher rate — a spill-over that could be more damaging than a brief read outage. Cross-region topologies need an explicit decision about whether falling back is even desirable, which is covered in designing multi-region read replica topologies.

FAQ

How much does cross-zone database traffic actually cost?

It is charged per gigabyte in both directions on most major providers, and read result sets are usually the largest database traffic component by volume. The figure worth computing is your own: multiply the measured bytes returned by read queries per month by the per-gigabyte cross-zone rate. For a read-heavy service returning tens of terabytes a month it is frequently a larger line item than the replica instances themselves; for a small service it is negligible — measure before optimising.

Does zone-aware routing hurt load balance?

Yes, deliberately. It makes traffic follow your application tier’s zone distribution rather than spreading evenly, so a zone hosting 40% of your application instances sends 40% of the reads to its local replicas. That is acceptable only if each zone’s replicas can absorb their zone’s share, which is a capacity question to answer before enabling it rather than after.

What is the latency benefit of staying in-zone?

Typically a few hundred microseconds to about a millisecond of round-trip time per query. That sounds negligible until you count queries: a request that issues thirty sequential reads pays it thirty times, so one millisecond becomes thirty on the request’s critical path. The benefit is proportional to how chatty your read path is — large for ORM-heavy code, small for services that issue one aggregate query.

Should the primary also be zone-aware?

No — there is only one primary, so writes cross zones whenever the caller is not in its zone and nothing can be done about that short of moving the primary. Zone awareness applies only to the read path, where multiple equivalent targets exist. Attempting to co-locate application instances with the primary to avoid it creates a much worse dependency on that one zone.

← Back to Load Balancing Reads Across a Replica Pool