What Happens to Read Replicas During an RDS Multi-AZ Failover

Problem statement: a Multi-AZ failover is in progress on a managed RDS instance, you cannot influence the promotion at all, and you need to know exactly what your read replicas will do so the read path can be designed to survive it.

The managed case inverts the usual problem. With a self-managed cluster you control the promotion and worry about doing it correctly; with RDS the promotion is a black box and the entire engineering problem is the observable behaviour at your side of the boundary. This page catalogues that behaviour and what to do about each part of it. The trade-offs of ceding this control are covered in managed vs self-managed read replicas.

Symptom Identification

During and shortly after a Multi-AZ failover you will observe, in roughly this order:

  • Writes fail for tens of seconds. Connections to the writer endpoint are dropped; new connections fail or connect to an instance that is not yet accepting writes.
  • Reads keep succeeding — from frozen data. Read replicas stay open and readable. pg_last_xact_replay_timestamp() stops advancing, so ReplicaLag climbs linearly at one second per second while every health check reports the replica healthy.
  • A ReplicaLag spike after recovery. When replication resumes from the promoted standby, replicas catch up in a burst. The metric spikes, then falls sharply as the backlog is applied.
  • Some clients never recover. A subset of application instances continue to fail long after the endpoint has been repointed, because their DNS cache still holds the old address.
  • RDS events record the sequence. describe-events shows Multi-AZ instance failover started and completed, with replica-side Replication for the Read Replica resumed entries following.

The second symptom is the one that matters most and gets the least attention. A read tier that appears completely healthy while serving data frozen at the moment of failover is far more dangerous than one that fails visibly.

Root Cause Analysis

The Multi-AZ standby is not one of your read replicas. RDS Multi-AZ maintains a hidden standby in another zone that exists only to be promoted. It serves no reads and is invisible in the console except as a zone attribute. Your read replicas are separate instances with their own endpoints, and they are never promotion candidates in a Multi-AZ event.

The failover changes the replicas’ source, not their role. After promotion the read replicas must resume replication from the newly promoted instance. AWS handles the repointing, but there is a window between “the old source stopped” and “the new source is streaming” during which the replica has nothing to apply. It stays up, stays readable, and stops moving.

The writer endpoint is a DNS record, and DNS is where the time goes. The failover itself completes in tens of seconds. The record is updated promptly. What takes longer is every client’s cache expiring. A JVM configured with the default networkaddress.cache.ttl under some security managers caches successful lookups for the lifetime of the process, which means those instances never recover without a restart.

The lag spike is a consequence of the gap, not of a problem. When replication resumes, the replica applies the accumulated backlog as fast as it can. ReplicaLag peaks at roughly the duration of the gap and then decays. Alerting configured with a short for duration will fire on this every single time, training everyone to ignore the alert.

Observable behaviour across a Multi-AZ failover, by component A shared time axis with three lanes. The writer lane shows connections dropping at failover, the endpoint DNS record being repointed within about thirty seconds, and writes resuming. The read replica lane shows reads continuing throughout, with replayed data frozen during the gap and then catching up in a burst once the new source is attached. The client lane shows one application instance recovering promptly and another continuing to fail because its resolver cache still holds the old address. t0 +30 s +60 s +120 s writer endpoint connections dropped DNS swap writes accepted again read replica stays up throughout reads served the whole time replay frozen — data ages, health checks pass catch-up burst, ReplicaLag spikes then falls clients errors TTL 5 s — recovers with the endpoint TTL cached indefinitely — still failing, needs a restart The database recovered at +40 s. Whether your users did depends entirely on the bottom lane.

Step-by-Step Resolution

Step 1 — Cap DNS caching in every runtime

java
// JVM: the single highest-impact change for RDS failover time.
// Without this, a successful lookup can be cached for the process lifetime.
java.security.Security.setProperty("networkaddress.cache.ttl", "5");
java.security.Security.setProperty("networkaddress.cache.negative.ttl", "1");
python
# Python and most C-based runtimes honour the OS resolver, which respects
# the record's own TTL. The risk there is a connection pool holding an
# already-resolved socket — bound the connection lifetime instead.
POOL = {"max_lifetime_s": 300, "connect_timeout_s": 3}

Inline verification: during a forced failover, confirm from application logs that the first successful reconnect occurs within seconds of the RDS failover completed event, not minutes.

Step 2 — Keep reads off the writer endpoint

Reads that resolve through the writer endpoint inherit every second of the writer’s outage for no benefit. Point them at the individual replica endpoints, or at a load balancer over them:

yaml
# Read and write paths must not share an endpoint. The writer endpoint
# is a failover-managed record; the replica endpoints are stable instances.
databases:
  writer:  "app-prod.cluster-abc123.eu-west-1.rds.amazonaws.com"
  readers:
    - "app-prod-replica-1.abc123.eu-west-1.rds.amazonaws.com"
    - "app-prod-replica-2.abc123.eu-west-1.rds.amazonaws.com"

Inline verification: during a forced failover, read query rate stays flat while write query rate drops to zero — proof the two paths are genuinely independent.

Step 3 — Make the driver give up on a stalled socket quickly

A failover drops connections without a clean close in some paths, so a client can wait on the OS default TCP timeout — minutes — before noticing.

ini
# libpq / psycopg connection parameters
connect_timeout=3
keepalives=1
keepalives_idle=10
keepalives_interval=3
keepalives_count=3      # dead socket detected in ~19 s, not ~15 min
tcp_user_timeout=15000  # cap on unacknowledged transmission, in ms

Inline verification: ss -tio state established '( dport = :5432 )' shows the keepalive timer active on established connections.

Step 4 — Absorb the lag spike rather than alerting on it

yaml
# CloudWatch alarm tuned so a normal post-failover catch-up does not page.
# EvaluationPeriods × Period must exceed the catch-up duration you measured.
AlarmName: ReplicaLagSustained
MetricName: ReplicaLag
Namespace: AWS/RDS
Statistic: Maximum
Period: 60
EvaluationPeriods: 5           # 5 minutes sustained, not a single sample
Threshold: 30
TreatMissingData: breaching    # a replica reporting nothing is not healthy

The TreatMissingData: breaching setting matters here: during the gap the replica may stop publishing the metric entirely, and the default treatment would read that silence as “fine”.

Inline verification: after a rehearsal failover, confirm the alarm did not enter ALARM state for the catch-up spike but would have for a genuine sustained lag.

Step 5 — Rehearse and measure

bash
# Force a failover in a maintenance window. This is the only way to learn
# what your clients actually do.
aws rds reboot-db-instance --db-instance-identifier app-prod --force-failover

# Watch the event stream for the authoritative timeline.
aws rds describe-events --source-identifier app-prod --source-type db-instance \
    --duration 20 --query 'Events[].[Date,Message]' --output table

Inline verification: record three numbers — time to writer endpoint repointed, time to first successful client write, and peak ReplicaLag. The gap between the first two is your DNS problem, measured.

Configuration Snippet

hcl
# Terraform: the instance-level settings that shape failover behaviour.
resource "aws_db_instance" "primary" {
  identifier           = "app-prod"
  multi_az             = true          # creates the hidden standby
  backup_retention_period = 7          # required for read replicas to exist at all

  # Apply parameter changes in the maintenance window, not immediately —
  # an immediate apply on a Multi-AZ instance can itself trigger a failover.
  apply_immediately    = false
  maintenance_window   = "Sun:03:00-Sun:04:00"
}

resource "aws_db_instance" "replica" {
  count               = 2
  identifier          = "app-prod-replica-${count.index + 1}"
  replicate_source_db = aws_db_instance.primary.identifier

  # Place replicas in zones other than the primary's, so a zone event does
  # not remove the primary and the whole read tier together.
  availability_zone   = element(["eu-west-1b", "eu-west-1c"], count.index)

  # Replicas do not inherit the source's parameter group; set it explicitly
  # or they silently run defaults for max_standby_streaming_delay and friends.
  parameter_group_name = aws_db_parameter_group.replica.name
}

resource "aws_db_parameter_group" "replica" {
  name   = "app-prod-replica-pg16"
  family = "postgres16"

  # Give replica queries room to finish before recovery conflicts cancel them.
  # The catch-up burst after a failover is exactly when conflicts cluster.
  parameter {
    name  = "max_standby_streaming_delay"
    value = "30000"          # ms
  }
  parameter {
    name  = "hot_standby_feedback"
    value = "1"              # fewer cancellations, at the cost of primary bloat
  }
}

The replica parameter group is easy to overlook and directly shapes what users see during the catch-up burst. When a replica applies a large backlog quickly, recovery conflicts with long-running read queries become likely, and the default max_standby_streaming_delay cancels those queries — producing a wave of errors after the failover has otherwise completed. The mechanism and its trade-offs are covered in detecting hot standby query cancellations on Postgres replicas.

Failover time you control versus failover time AWS controls A stacked bar broken into segments. The AWS-controlled promotion and endpoint repoint occupy a small fixed share at the left. The client-controlled segments — resolver cache expiry, stale socket detection, and connection pool refresh — occupy the remaining and much larger share. A second bar below shows the same failover after client-side tuning, where the client segments shrink to a fraction and total observed outage falls by roughly two thirds. observed outage, before client tuning AWS: promote DNS set client resolver cache expiry stale socket timeout ≈ 4 min same failover, TTL 5 s and keepalives set AWS: promote DNS set ≈ 50 s — the AWS part is now most of it AWS-controlled — you cannot shorten this client-controlled — you can Every minute of a managed failover that you can actually remove lives on the client, not in the console.

Verification and Rollback

Confirm the replicas reattached and caught up:

sql
-- On each read replica.
SELECT pg_is_in_recovery()                                 AS still_a_replica,
       now() - pg_last_xact_replay_timestamp()             AS replay_age,
       pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() AS fully_caught_up;
bash
# The provider's own view, which is what your alarms read.
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS --metric-name ReplicaLag \
  --dimensions Name=DBInstanceIdentifier,Value=app-prod-replica-1 \
  --start-time "$(date -u -d '30 min ago' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" --period 60 --statistics Maximum

A replica whose replay_age keeps growing an hour after the failover has not reattached. RDS usually recovers this itself; if it does not, the remedy is a reboot of the replica instance, and if that fails, recreating it — there is no equivalent of standby follow available to you.

Confirm the clients recovered:

sql
-- On the new primary: are all your application hosts back?
SELECT client_addr, count(*) FROM pg_stat_activity
 WHERE backend_type = 'client backend'
 GROUP BY client_addr ORDER BY 2 DESC;

Compare the address list against the expected fleet. A missing host is a client still holding a stale resolution, and it will not recover on its own.

Rollback. There is nothing to roll back in the failover itself — it cannot be undone, and the promoted instance is now the primary. What can be reversed is a configuration change made in response: if shortening DNS TTL or keepalive intervals causes unexpected churn, revert those independently. Note that RDS failovers alternate zones, so a subsequent forced failover returns the primary to the original zone if zone placement matters for your latency budget.

Edge Cases and Gotchas

A replica in the same zone as the primary is a correlated risk

If a read replica shares a zone with the primary, a zone-level event removes both — you lose write availability and a third of your read capacity simultaneously, exactly when the fallback path is most needed. Spread replicas across the zones the primary is not in, which is why the Terraform snippet above sets availability_zone explicitly rather than letting AWS choose.

Cross-region replicas experience a much longer gap

A cross-region read replica must re-establish a wide-area replication connection after the source changes, and the accumulated backlog crosses that link at whatever bandwidth is available. Where a same-region replica catches up in seconds, a cross-region one can take minutes. Plan the read path so a cross-region replica is not the fallback target for freshness-sensitive reads during a failover window — the cost model is discussed in cross-region read replicas on RDS: cost and lag trade-offs.

Applying a parameter change immediately can itself cause a failover

Setting apply_immediately = true on a Multi-AZ instance for a parameter that requires a reboot will reboot with failover. Teams discover this when a routine parameter tweak becomes an unplanned failover in the middle of the day. Use the maintenance window unless the change is genuinely urgent, and know which parameters are static before applying them.

FAQ

Are RDS read replicas promoted during a Multi-AZ failover?

No. A Multi-AZ failover promotes the hidden standby that exists solely for that purpose; your read replicas are not candidates and remain replicas throughout. What happens to them is that their replication source changes underneath them, which causes a brief disconnection and a lag spike but no role change and no promotion.

Why did my application keep connecting to the old primary after the failover?

Client-side DNS caching. The writer endpoint’s DNS record is repointed within seconds, but a runtime that caches resolutions — the JVM caches successful lookups for the process lifetime under some security policies — keeps using the old address until its cache expires. This is the single largest contributor to observed failover time and it is fixed entirely on the client.

Do read replicas keep serving reads during the failover?

Yes, mostly. A read replica remains open and readable while its source fails over, so reads continue to be served — from data frozen at the moment the source disappeared, then jumping forward when replication resumes. The risk is not unavailability but unnoticed staleness, so a freshness gate matters more here than a health check.

Should reads use the writer endpoint as a fallback?

Only deliberately and with a bound. Falling back to the writer during a replica outage protects correctness but concentrates read load on the instance already recovering from a failover, which is exactly when it has the least headroom. Cap the fallback share, or degrade the read path instead, and decide which before the incident rather than during it.

← Back to Replica Failover & Promotion Mechanics