Monitoring & Observability for Read Replicas

A read replica fleet fails quietly. Unlike a crashed primary, a replica that has fallen minutes behind, silently disconnected its replication stream, or exhausted its connection pool keeps answering queries — it just answers them with stale or slow data, and your application has no way to know unless you have instrumented the gap. Observability for read replicas is the discipline of turning that invisible degradation into signals you can alert on, budget against, and route around before a customer sees a stale balance or a timed-out dashboard. This guide maps the full production surface: what to measure on each node, how to move those measurements through an exporter-to-Prometheus-to-Grafana-and-Alertmanager pipeline, how to frame replica staleness as a service-level objective with an error budget, and where the standard monitoring setups develop blind spots that let a broken replica pass for a healthy one. It sits alongside the work of actually detecting and handling replication lag in real time; this page is the measurement and telemetry foundation that lag handling, routing, and pooling all consume.

Architecture Overview

Every observable signal about a replica originates inside the database process and has to travel through a fixed pipeline before it becomes an alert on someone’s phone or a routing decision at the proxy. The diagram below traces that path: exporters translate internal database counters into a scrapeable format, Prometheus pulls and stores them, recording rules precompute the expensive derivations, and the two consumers — Grafana for human eyes and Alertmanager for automated response — fan out to dashboards and on-call.

Read replica metrics pipeline from database nodes through exporters to Prometheus, Grafana, and Alertmanager A primary database, two hot-standby replicas, and a PgBouncer pooler each run an exporter that exposes internal counters. Prometheus scrapes every exporter on a fixed interval and evaluates recording and alerting rules against the stored series. Grafana queries Prometheus to render fleet dashboards, and Alertmanager receives fired alerts and routes them to on-call and to the query router that ejects lagging replicas. Primary WAL sender / binlog Replica A apply / replay Replica B apply / replay PgBouncer pool for replicas postgres_exporter reads pg_stat_replication postgres_exporter replay_lsn / heartbeat postgres_exporter replay_lsn / heartbeat pgbouncer_exporter cl_waiting / sv_active Prometheus scrape every 10s TSDB storage recording rules alerting rules pull scrape → Grafana fleet dashboards Alertmanager dedupe / route PromQL fired alerts On-call / query router page + eject replica operator

The most important property of this pipeline is that it is pull-based and layered. Prometheus reaches out to scrape each exporter; the exporter is not pushing. That means an exporter or a whole node can go dark without any error propagating forward — the absence of data is itself the signal, and you must alert on that absence explicitly (see Failure Modes below). The second property is that every derived quantity you care about — lag in seconds, pool saturation as a ratio, query latency percentiles — is computed after collection, either in a recording rule or at query time, from raw counters the exporter exposes. Getting the raw counters right at the source is therefore non-negotiable; no amount of dashboard polish recovers a signal that was never collected.

What to Measure

A read replica is healthy along several independent axes, and a single number never captures all of them. The signal catalog below is the minimum viable set for a production fleet. Each maps to a concrete counter you can export.

Replication lag, measured two ways. The byte delta between the primary’s current write position and the replica’s replay position — pg_wal_lsn_diff(sent_lsn, replay_lsn) from pg_stat_replication on PostgreSQL, or the GTID gap on MySQL — tells you how much data is queued but not yet applied. The wall-clock delta tells you how old the replica’s view is. On PostgreSQL, EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) gives replay lag in seconds; on MySQL, Seconds_Behind_Source from SHOW REPLICA STATUS is the rough equivalent. Byte lag and time lag answer different questions and can diverge sharply: a replica can be a few kilobytes behind but seconds stale if apply is stalled, or megabytes behind but sub-second stale if it is streaming fast. Export both.

Heartbeat lag. Both Seconds_Behind_Source and pg_last_xact_replay_timestamp are derived from the last transaction the replica applied, so on an idle primary they read near-zero even when replication is broken. The robust measurement is a heartbeat: a background job writes now() into a dedicated row on the primary every second or two, and lag is now() - heartbeat_row_timestamp read from each replica. This keeps advancing whether or not application traffic is flowing, and it is the single most important lag signal to get right.

Apply-queue depth. On PostgreSQL streaming replication the receive and replay positions (pg_last_wal_receive_lsn versus pg_last_wal_replay_lsn) reveal WAL that has arrived but not yet been applied — a growing gap here means the single-threaded (or parallel) apply worker is the bottleneck, not the network. On MySQL the equivalent is the gap between Retrieved_Gtid_Set and Executed_Gtid_Set.

Pool saturation. If reads flow through a pooler, its own metrics are first-class. pgbouncer_exporter exposes cl_waiting (clients blocked waiting for a server connection), sv_active, sv_idle, and maxwait. A non-zero cl_waiting sustained over any interval means the pool is the constraint, not the database. This is the observability counterpart to sizing decisions covered in connection pool architecture for read replicas.

Hot-standby conflicts and query cancellations. PostgreSQL replicas cancel long-running read queries when they conflict with incoming WAL (a vacuum on the primary removing rows the query still needs). These show up in pg_stat_database_conflicts — confl_snapshot, confl_lock, confl_bufferpin. A rising conflict rate is a signal unique to replicas that no primary-side monitoring will ever surface, and it directly degrades read reliability.

Per-replica query latency. The same query can run at very different speeds on different replicas depending on cache warmth, apply pressure, and hardware. Track statement latency per node, ideally from pg_stat_statements, so you can see when one replica in the fleet is systematically slow.

Connection counts and states. pg_stat_activity broken down by state (active, idle, idle in transaction) per replica catches connection leaks and the “idle in transaction” pileups that silently consume the connection budget.

Metric Trade-off Matrix

Not every metric is worth collecting at the same frequency or trusting equally. Each has a cost to scrape and a characteristic way it lies. Choose the signal that best fits the question you are actually asking.

Metric What it detects Scrape cost False-positive risk Best-fit signal
Heartbeat lag (seconds) True wall-clock staleness, even when idle Low (one row read) Very low — advances regardless of traffic Primary lag SLO signal
WAL/LSN byte delta Volume of unapplied changes queued Low (from pg_stat_replication) Medium — large under bulk load without real staleness Apply-pipeline pressure
Seconds_Behind_Source / replay timestamp Apply lag from last applied txn Low High — reads zero on an idle primary Quick MySQL glance, not SLO
Apply-queue depth (recv−replay) Apply worker is the bottleneck Low Low Distinguishing network vs apply stalls
Pool saturation (cl_waiting, maxwait) Pooler is the constraint, not the DB Low Low Connection exhaustion
Hot-standby conflicts WAL-vs-query cancellations on standby Medium (per-database) Low Replica read reliability
Per-replica query latency (pg_stat_statements) One node systematically slow Medium–high (high cardinality) Medium — noisy on low sample counts Query performance regression
Connection state counts Leaks, idle-in-txn pileups Low Low Capacity and leak detection

The recurring trap in this table is the third row. Seconds_Behind_Source and its PostgreSQL equivalent are cheap and intuitive, which is exactly why teams build alerts on them and then get burned when the metric reads zero during an idle window while the replication link is actually dead. Treat those as convenience glances and anchor your actual objective to heartbeat lag.

The Metrics Pipeline

The collection layer is a small number of well-known exporters, each a sidecar or co-located process that connects to one database or pooler and exposes an HTTP /metrics endpoint in Prometheus text format.

  • postgres_exporter connects to a PostgreSQL instance and exposes replication, activity, and (with custom queries) heartbeat and conflict metrics. Point one at each replica and at the primary — you need the primary’s sent_lsn to compute byte lag.
  • mysqld_exporter is the MySQL equivalent, surfacing SHOW REPLICA STATUS, SHOW GLOBAL STATUS, and GTID sets.
  • pgbouncer_exporter connects to PgBouncer’s admin SHOW interface and exposes pool-level saturation counters.

These feed Prometheus, which scrapes on a fixed interval, stores the raw series in its local TSDB, and continuously evaluates recording and alerting rules. Grafana reads Prometheus over PromQL to render dashboards for human operators, and Alertmanager receives alerts that Prometheus fires, deduplicates and groups them, and routes to on-call — and, in mature setups, to the query router that ejects a lagging replica from the read pool automatically.

The critical design choice within this pipeline is between two complementary styles of measurement:

White-box monitoring reads the database’s own internal counters. This is everything discussed so far: WAL positions, apply queue, conflict counters, pool stats. It is rich and diagnostic — it tells you why a replica is degraded — but it trusts the database to report accurately about itself, and it goes blind the moment the exporter or the node dies.

Black-box monitoring probes the replica from the outside, exactly as an application would. A small prober opens a real connection to the replica endpoint, runs a canary query (for example, reading back a token it just wrote to the primary and waiting for it to appear), and measures whether the answer is correct and how long it took. Black-box monitoring makes no assumptions about the internals; it directly measures the user-visible symptom — “can I read fresh data from this replica right now?” — and it keeps working when white-box telemetry has gone dark.

Neither is sufficient alone. White-box tells you the apply queue is growing; black-box tells you that reads are already stale from the caller’s perspective. Run both, and treat a disagreement between them (healthy internals, failing probe, or vice versa) as its own high-signal alert. The routing decisions that consume these signals are covered across connection routing and pooling strategies and the real-time handling in replication lag and consistency management.

Configuration Baseline

Below is an annotated prometheus.yml scrape configuration for a fleet of PostgreSQL replicas plus a pooler, followed by the recording rule that turns raw counters into the derived lag-seconds series your dashboards and alerts should read.

yaml
# prometheus.yml — scrape config for a read-replica fleet
global:
  scrape_interval: 10s          # baseline; short enough to catch lag spikes
  scrape_timeout: 5s            # must be < scrape_interval
  evaluation_interval: 15s      # how often recording/alerting rules run

rule_files:
  - /etc/prometheus/rules/replica.rules.yml

scrape_configs:
  # postgres_exporter on the primary AND every replica.
  # The primary is required to compute byte lag (its sent_lsn).
  - job_name: postgres
    scrape_interval: 10s
    static_configs:
      - targets: ['primary:9187']
        labels: {role: primary, cluster: orders}
      - targets: ['replica-a:9187']
        labels: {role: replica, cluster: orders, replica: a}
      - targets: ['replica-b:9187']
        labels: {role: replica, cluster: orders, replica: b}

  # pgbouncer_exporter fronting the replica read pool.
  - job_name: pgbouncer
    scrape_interval: 10s
    static_configs:
      - targets: ['replica-pool:9127']
        labels: {cluster: orders, pool: replica_ro}

The heartbeat metric that this depends on is exported by a custom query on postgres_exporter that reads a heartbeat table the primary updates every second. The recording rule then converts it into a clean, cheap per-replica series:

yaml
# /etc/prometheus/rules/replica.rules.yml
groups:
  - name: replica_derived
    interval: 15s
    rules:
      # Wall-clock staleness from the heartbeat row — the SLO signal.
      # pg_replica_heartbeat_seconds is the epoch of the newest heartbeat
      # the replica has applied; subtract it from scrape time.
      - record: replica:heartbeat_lag_seconds
        expr: time() - pg_replica_heartbeat_seconds{role="replica"}

      # Byte lag between primary sent_lsn and each replica replay_lsn.
      - record: replica:wal_lag_bytes
        expr: |
          pg_replication_sent_lsn_bytes{role="primary"}
            - on(cluster) group_right
          pg_replication_replay_lsn_bytes{role="replica"}

      # Pool saturation as a ratio: waiting clients over pool size.
      - record: replica:pool_saturation_ratio
        expr: |
          pgbouncer_pools_client_waiting_connections
            / clamp_min(pgbouncer_pools_server_active_connections
              + pgbouncer_pools_server_idle_connections, 1)

Recording rules matter here for two reasons. First, dashboards and alerts read the precomputed replica:heartbeat_lag_seconds instead of re-deriving it on every query, which keeps Grafana responsive and alert evaluation cheap. Second, the derived series is where you name the signal once — every consumer references the same definition, so there is no drift between what the dashboard shows and what the alert fires on. The concrete exporter queries behind these metrics are the subject of Prometheus metrics for replica health and lag.

SLO & Error-Budget Framing for Replica Staleness

Alerting on raw lag (“page when lag exceeds 2 seconds”) produces noise, because a two-second spike that self-corrects in ten seconds is not an incident — but a two-second floor sustained for an hour is. The durable framing is a service-level objective on staleness.

Express the objective as the fraction of reads served within an acceptable staleness bound over a rolling window. For example: 99.9% of tier-1 reads are served by a replica with under 500 ms of heartbeat lag, measured over 30 days. The complement of the target — the 0.1% you are allowed to miss — is your error budget. Over 30 days that is roughly 43 minutes of out-of-bound reads. Framing it this way converts an anxious “is lag bad right now?” into a calm accounting question: “how much of this month’s budget have we spent, and how fast?”

Alert on the burn rate of that budget, not on instantaneous lag:

  • A fast-burn rule pages: it fires when the fleet is consuming the monthly budget so quickly that a large fraction would be gone within an hour (for example, burning at 14× the sustainable rate over a 1-hour window). This is a genuine, act-now incident.
  • A slow-burn rule opens a ticket: it fires on sustained low-grade staleness that would exhaust the budget over days if left alone — a replica that is chronically 300 ms behind, not catastrophically broken but steadily eating margin.

This two-tier approach is what keeps on-call sane: transient spikes never page, sustained degradation always surfaces, and the severity matches the actual customer impact. The alert rules that implement burn-rate windows are detailed in alerting on replication lag and pool saturation. The staleness bounds themselves — what “acceptable” means per tier — come from the consistency requirements analyzed in replication lag and consistency management.

Failure Modes

Replica monitoring fails in characteristic ways, and every one of them shares a property: the dashboard looks green while the system is degraded. These are the blind spots to design against explicitly.

Scraping the wrong node. The most common silent failure is pointing the “replica lag” panel at the primary, or at a replica that was promoted and is no longer replicating. On the primary, pg_last_wal_replay_lsn() and heartbeat lag return healthy-looking values that mean nothing about any actual standby. Guard against it by labelling every target with its role and asserting pg_is_in_recovery() is true on nodes you treat as replicas — alert if a target labelled replica reports it is no longer in recovery.

A lag metric that lies during idle. As covered above, Seconds_Behind_Source and replay-timestamp lag read zero when the primary stops committing. An alert built only on that metric will happily report zero lag across a replica whose replication link has been dead for an hour, because there is simply nothing newer for it to be behind on. The heartbeat metric is the fix; if you are alerting on replay timestamp alone, you have this bug latent in your setup right now.

The exporter dying silently. Because Prometheus pulls, a dead exporter produces absence of data, not an error. Panels go flat, and a naive alert that checks lag > threshold never fires because there is no series to evaluate. Every scrape target needs an up == 0 and an absent() alert so that a missing exporter pages instead of masquerading as health. This is the single most important alert in the whole setup and the one most often missing.

Cardinality blowups. Per-replica, per-query metrics from pg_stat_statements are diagnostic gold and a cardinality landmine. Exporting a distinct time series for every normalized query fingerprint across every replica can multiply into millions of series, blow up Prometheus memory, and slow every query — including the ones your alerts depend on. Bound it: cap the number of statements exported per node (top-N by total time), drop high-churn label values, and keep raw statement text out of labels entirely. The techniques for doing this safely are covered in query performance analysis on read replicas.

Scrape interval longer than the spike. A 60-second scrape interval cannot see a 5-second lag spike that breaches your SLO and recovers before the next scrape. Match the scrape interval to the resolution of the smallest event you must catch; for tier-1 lag SLOs that usually means 10 seconds or tighter.

What This Section Covers

The guides in this section go deep on each stage of the pipeline above — collecting the right metrics, visualizing them, alerting on them, and analyzing query performance on the replicas themselves.

Prometheus Metrics for Replica Health & Lag The collection foundation: the exact postgres_exporter and mysqld_exporter custom queries that expose heartbeat lag, WAL/LSN byte delta, apply-queue depth, and hot-standby conflicts. Covers how to structure the heartbeat table, how to label series for a multi-replica fleet, and how to monitor pool saturation with pgbouncer_exporter so that every downstream dashboard and alert reads a correct, correctly-labelled series.

Grafana Dashboards for Read Replica Fleets How to turn the collected series into a fleet view an operator can read at a glance: lag distribution across every replica, apply-queue depth over time, connection-state breakdowns, and per-node query latency. Covers template variables for scaling one dashboard across many clusters, and building a lag heatmap that makes an outlier replica jump out of a fleet of dozens.

Alerting on Replication Lag & Pool Saturation The alerting layer: burn-rate rules that page on fast SLO exhaustion and ticket on slow burn, plus the up == 0 and absent() rules that catch a dead exporter. Covers Alertmanager routing and grouping so that a whole-fleet lag event produces one page rather than fifty, and how to wire pool-saturation alerts that fire before clients start timing out.

Query Performance Analysis on Read Replicas Finding and fixing slow reads on the replicas themselves: using pg_stat_statements per node without triggering a cardinality blowup, detecting hot-standby query cancellations, and distinguishing a query that is slow everywhere from one that is slow only on a specific lagging replica. Covers the interaction between apply pressure and read latency that makes replica query tuning distinct from primary tuning.

Operational Checklist

FAQ

What is the difference between black-box and white-box replica monitoring?

Black-box monitoring probes a replica from the outside as a user would: it opens a connection, runs a canary query, and measures whether the answer is correct and timely. White-box monitoring reads the database’s own internal counters through an exporter: WAL positions, apply-queue depth, buffer cache hit ratio, lock waits. Black-box tells you the replica is broken; white-box tells you why. Production fleets need both because each catches failures the other misses — a dead exporter blinds white-box while black-box still sees the stale read, and a subtle apply stall shows in white-box counters before it grows large enough for a probe to notice.

Why does Seconds_Behind_Source read zero when a replica is actually stale?

Seconds_Behind_Source and pg_last_xact_replay_timestamp are derived from the timestamp of the last transaction the replica applied. On an idle primary that stops committing, the replica has nothing new to apply, so the metric reads zero or near-zero even though the replication link may be broken. The fix is a heartbeat: write a timestamp row to the primary on a fixed interval and measure lag as the delta between now and the replica’s copy of that row, which keeps advancing even when application traffic is idle.

How do I set an SLO for replica staleness?

Express the SLO as the fraction of read requests served by a replica within an acceptable staleness bound over a rolling window, for example 99.9 percent of tier-1 reads served with under 500 ms of lag over 30 days. The complement of the target is the error budget. Alert on the burn rate of that budget rather than on instantaneous lag: a fast-burn rule pages when a large fraction of the monthly budget is consumed in an hour, and a slow-burn rule opens a ticket for sustained low-grade staleness.

← Back to Home