Alerting on Replication Lag and Pool Saturation

An alert that pages at 3 a.m. for a lag spike that self-corrected before the responder opened their laptop is worse than no alert at all — it erodes trust in the whole system. Alerting on read replicas has two failure modes that pull in opposite directions: alert too eagerly and you drown responders in flapping noise; alert too conservatively and a genuinely stale replica silently serves wrong data for minutes. This section builds the alerting layer that sits on top of your Prometheus metrics for replica health and lag, turning raw time series into a small set of high-signal pages that map directly to a runbook. It is part of the broader monitoring and observability practice for read replicas.

The two quantities that dominate replica incidents are replication lag — how far behind the primary a replica’s applied state is — and pool saturation — whether the connection pool fronting the replica has run out of usable backend connections. Both degrade read correctness or availability, both have well-defined telemetry, and both are prone to false alarms if you alert on instantaneous values. The design goal is a rule set where every firing alert is actionable, correctly attributed, and de-duplicated against upstream causes.

Concept Definition and Scope

Alerting is the decision layer that converts continuously scraped metrics into discrete, routed notifications. It has three moving parts, and confusing them is the root of most bad alerting:

1. Alert rules live in Prometheus. Each rule is a PromQL expression evaluated on a fixed interval; when the expression returns a non-empty vector for longer than the rule’s for: duration, the alert transitions from pending to firing.
2. Routing and grouping live in Alertmanager. Firing alerts are grouped by label, deduplicated, throttled, and dispatched to receivers (PagerDuty, Slack, email, webhook).
3. Inhibition and silencing suppress alerts that are redundant (inhibited by an upstream cause) or expected (silenced during a maintenance window).

This page covers all three for the two signal families — lag and saturation — plus the meta-signal of a dead exporter, and the multi-window multi-burn-rate pattern for read-staleness SLOs. The deep dive on SLO burn-rate math and the full rule YAML lives in the companion guide, Alertmanager rules for replica lag SLO burn rate. What is out of scope here: dashboarding (a visualization concern), and the exporter queries that produce the metrics — those belong to the metrics collection guide.

Mechanism Deep-Dive

An alert fires through a fixed pipeline: Prometheus evaluates the rule expression on its evaluation_interval; a matching sample starts a for: timer; once the timer elapses the alert is pushed to Alertmanager, which groups it, applies inhibition, and dispatches it to the matched receiver. Understanding where each control lives tells you which knob to turn when an alert misbehaves.

The three-part lag alert

A production-grade lag alert combines three conditions so it catches both slow drift and fast runaways without flapping on noise:

groups:
  - name: replica_lag
    rules:
      # 1. Absolute threshold with a sustained hold to defeat flapping
      - alert: ReplicaLagHigh
        expr: pg_replication_lag_seconds{role="replica"} > 30
        for: 5m
        labels:
          severity: warning
          cluster: "{{ $labels.cluster }}"
        annotations:
          summary: "Replica {{ $labels.instance }} lag > 30s for 5m"
          runbook_url: "https://runbooks.internal/db/replica-lag-high"
      # 2. Rate-of-change: lag climbing fast enough to breach soon
      - alert: ReplicaLagClimbing
        expr: |
          deriv(pg_replication_lag_seconds{role="replica"}[5m]) > 0.5
          and pg_replication_lag_seconds{role="replica"} > 10
        for: 3m
        labels:
          severity: warning
          cluster: "{{ $labels.cluster }}"
        annotations:
          summary: "Replica {{ $labels.instance }} lag rising >0.5s/s"
          runbook_url: "https://runbooks.internal/db/replica-lag-climbing"
      # 3. Critical absolute breach — a replica this far behind serves wrong data
      - alert: ReplicaLagCritical
        expr: pg_replication_lag_seconds{role="replica"} > 120
        for: 2m
        labels:
          severity: critical
          cluster: "{{ $labels.cluster }}"
        annotations:
          summary: "Replica {{ $labels.instance }} lag > 120s — pull from read pool"
          runbook_url: "https://runbooks.internal/db/replica-lag-critical"

The for: field is the flapping defense. pg_replication_lag_seconds is inherently jittery — a single slow apply cycle can spike it for one scrape and recover on the next. Requiring the condition to hold for 5m means the alert only fires on sustained degradation, not on the sawtooth noise of normal WAL replay. The deriv() rule is the early-warning half: it fires when lag is climbing at more than 0.5 seconds per wall-clock second while already above a 10-second floor, catching a runaway before it crosses the absolute threshold and giving the responder lead time.

The pool saturation alert

Saturation is a different signal with a different shape. For PgBouncer, the authoritative metric is pgbouncer_pools_client_waiting_connections (cl_waiting): clients queued because every backend server connection is busy. A sustained non-zero value means the pool is the bottleneck, not the replica.

      # cl_waiting sustained above zero = clients starved for a backend connection
      - alert: PgBouncerPoolSaturated
        expr: pgbouncer_pools_client_waiting_connections{database=~".*_ro"} > 0
        for: 2m
        labels:
          severity: warning
          cluster: "{{ $labels.cluster }}"
        annotations:
          summary: "PgBouncer {{ $labels.instance }} pool {{ $labels.database }} has waiting clients"
          runbook_url: "https://runbooks.internal/db/pool-saturation"
      # maxwait: any single client waited longer than the acquisition budget
      - alert: PgBouncerMaxWaitHigh
        expr: pgbouncer_pools_client_maxwait_seconds{database=~".*_ro"} > 1
        for: 1m
        labels:
          severity: critical
          cluster: "{{ $labels.cluster }}"
        annotations:
          summary: "PgBouncer {{ $labels.instance }} client waited > 1s for a connection"
          runbook_url: "https://runbooks.internal/db/pool-maxwait"

cl_waiting > 0 sustained for 2m warns that queueing has become chronic. maxwait_seconds > 1 is the sharper critical signal: even one client waiting over a second means the acquisition budget is blown and user-facing latency is already degraded. Full derivation of these pool signals is in the PgBouncer pool saturation metrics guide.

The dead-exporter meta-alert

Every threshold alert has a blind spot: if the exporter stops responding, the metric goes stale and the alert cannot fire. A replica could be catastrophically lagged while your dashboards show the last-known-good value. Guard against this with two rules:

      - alert: ReplicaExporterDown
        expr: up{job=~"postgres_exporter|pgbouncer_exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Exporter {{ $labels.instance }} is down — lag alerts are blind"
          runbook_url: "https://runbooks.internal/db/exporter-down"
      - alert: ReplicaMetricAbsent
        expr: absent(pg_replication_lag_seconds{role="replica"})
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "No replica lag metric is being reported at all"

up == 0 catches a scrape target that is failing to respond; absent() catches the subtler case where the series disappears entirely (relabeling change, exporter crash, target dropped from service discovery). Without these, a monitoring gap masquerades as a healthy fleet.

Trade-Off Comparison Table

Different alert conditions trade sensitivity against noise. Choosing the right mix per signal is the core design decision.

Absolute thresholds are simple and defensible but say nothing about whether a breach matters relative to your error budget — that is what the burn-rate approach adds. Rate-of-change buys lead time at the cost of occasional false positives during legitimate catch-up bursts. In practice a mature fleet runs all of these together, differentiated by severity so only the budget-threatening ones page.

Configuration Runbook

PostgreSQL streaming replica lag rule

The canonical PostgreSQL lag metric is derived from pg_last_xact_replay_timestamp(). Alert on the exporter-exposed series rather than re-querying in PromQL:

- alert: PostgresReplicaLagHigh
  expr: |
    (
      pg_replication_lag_seconds{role="replica"} > 30
    )
    unless on (cluster)
    (
      pg_replication_primary_down{} == 1
    )
  for: 5m
  labels:
    severity: warning
  annotations:
    runbook_url: "https://runbooks.internal/db/pg-replica-lag"

The unless on (cluster) clause is a rule-level guard that suppresses lag alerts when the primary is known down — a belt-and-braces complement to Alertmanager inhibition, useful when you want the alert to never even reach firing state.

MySQL replica lag rule

For MySQL, the equivalent metric from mysqld_exporter is mysql_slave_status_seconds_behind_source (older exporters expose ..._seconds_behind_master):

- alert: MySQLReplicaLagHigh
  expr: mysql_slave_status_seconds_behind_source > 30
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "MySQL replica {{ $labels.instance }} is {{ $value }}s behind source"
    runbook_url: "https://runbooks.internal/db/mysql-replica-lag"
- alert: MySQLReplicationStopped
  expr: mysql_slave_status_slave_sql_running == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "MySQL SQL apply thread stopped on {{ $labels.instance }}"

Seconds_Behind_Source reports NULL (scraped as the series being absent or the SQL thread flag going to 0) when replication is broken — so pair the lag alert with an explicit slave_sql_running == 0 check, otherwise a fully stopped replica reports zero lag and never alerts.

Alertmanager routing, grouping, and inhibition

The Alertmanager config is where redundant alerts get collapsed. Group by cluster so all replicas in one cluster share a single notification, route by severity, and inhibit downstream alerts when the primary is down:

route:
  group_by: ['cluster', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: chat-info
  routes:
    - matchers: [severity="critical"]
      receiver: pager-critical
      group_wait: 10s
    - matchers: [severity="warning"]
      receiver: ticket-warning

inhibit_rules:
  # A primary outage makes every replica lag — suppress the replica noise
  - source_matchers: [alertname="PrimaryDown"]
    target_matchers: [alertname=~"ReplicaLag.*|PgBouncerPool.*"]
    equal: ['cluster']
  # A critical lag alert should mute the warning-level one for the same replica
  - source_matchers: [alertname="ReplicaLagCritical"]
    target_matchers: [alertname="ReplicaLagHigh"]
    equal: ['instance']

receivers:
  - name: pager-critical
    pagerduty_configs:
      - routing_key: "<pd-key>"
  - name: ticket-warning
    webhook_configs:
      - url: "http://ticketer.internal/alert"
  - name: chat-info
    slack_configs:
      - channel: "#db-alerts"
        api_url: "<slack-webhook>"

The first inhibition rule is the single most valuable line in this config: when a PrimaryDown alert fires, every replica in the deployment will show climbing lag because WAL streaming has stopped. Those are symptoms, not incidents. equal: ['cluster'] scopes the suppression so a primary outage in cluster A does not mute genuine lag in cluster B. The second rule prevents a replica from paging twice — once at warning, once at critical — for the same underlying breach.

Monitoring and Alerting Signals

The alerting layer itself needs to be observable. Track these to know your alerts are healthy:

• ALERTS{alertstate="pending"} — a rule stuck in pending and never reaching firing usually means the for: duration is longer than the condition’s typical lifetime; tune it down.
• prometheus_notifications_dropped_total — a rising counter means Alertmanager is unreachable and pages are being lost silently. Alert on this.
• alertmanager_notifications_failed_total{integration=...} — per-receiver delivery failures; a broken PagerDuty key surfaces here.
• Alert volume per receiver per week — if the pager fires more than a handful of times a week for lag, your thresholds are too tight or your replicas are genuinely under-provisioned.
• Time-in-firing distribution — alerts that consistently resolve within a minute of firing are flapping despite the for:; widen the window.

For read-staleness specifically, the highest-signal alert is not a raw threshold at all but a multi-window multi-burn-rate rule that measures how fast you are consuming your staleness error budget. Fast-burn (1-hour and 5-minute windows) pages immediately for acute degradation; slow-burn (6-hour and 30-minute windows) opens a ticket for a chronic drift that will exhaust the budget over a day. That construction, including the PromQL and the burn-rate arithmetic, is covered in full in the SLO burn-rate rules guide.

Failure Modes and Recovery Steps

1. Alert storm when the primary fails

Symptom: The primary goes down and dozens of pages arrive within seconds — every replica lag alert, every pool alert, every SLO burn alert firing at once.

Root cause: No inhibition rule tying replica-side symptoms to the primary-down cause. When the primary stops streaming WAL, all replicas’ lag climbs simultaneously.

Recovery:

1. Add the PrimaryDown inhibition rule shown above, keyed on equal: ['cluster'].
2. Emit a PrimaryDown alert from a health probe against the primary itself, not derived from replica state, so it fires first and inhibits the rest.
3. Verify by killing the primary in staging and confirming exactly one page arrives. When the primary recovers, the fallback behavior for the read tier should follow your fallback strategy for when replicas fall behind.

2. Flapping lag alert trains responders to ignore it

Symptom: ReplicaLagHigh fires and resolves several times an hour; on-call has muted the channel.

Root cause: for: too short (or absent) relative to the natural jitter of pg_replication_lag_seconds, so transient apply-cycle spikes cross the threshold.

Recovery:

1. Raise for: to at least 5m for warning-severity lag.
2. Replace the raw gauge with avg_over_time(pg_replication_lag_seconds[2m]) in the expression to smooth single-scrape spikes.
3. Move fast-detection responsibility to the deriv() rule or the burn-rate rule, which are designed to distinguish a real ramp from noise.

3. Silent replica while dashboards look green

Symptom: A replica served stale data for ten minutes but no alert fired; the lag graph flatlined at its last value.

Root cause: The exporter crashed or was dropped from service discovery; the lag series went stale and threshold rules cannot fire on a series that is not updating.

Recovery:

1. Deploy the up == 0 and absent() meta-alerts at critical severity.
2. Set Prometheus --query.lookback-delta awareness: a stale series stops matching threshold expressions after the lookback window, which is exactly why absent() is needed to catch it.
3. Route the exporter-down alert to the pager, not to chat — a blind monitoring stack is a critical condition.

4. Pool saturation misdiagnosed as replica lag

Symptom: Read latency spikes and the team spends an hour investigating WAL apply, but replicas are healthy.

Root cause: The alert fired on a generic “reads are slow” symptom without distinguishing pool queueing (cl_waiting) from apply lag (replication_lag_seconds).

Recovery:

1. Keep the two signal families in separate alerts with distinct summary text and distinct runbooks.
2. Add pgbouncer_pools_server_active_connections and ..._server_idle_connections to the saturation runbook so the responder can immediately see whether backends are maxed out.
3. If saturation is chronic, resize default_pool_size or shed load — do not tune replication.

Continue in This Topic Area

This section has one focused deep-dive guide:

Alertmanager Rules for Replica Lag SLO Burn Rate Defines a read-staleness SLO, derives the error budget, and walks through the full multi-burn-rate PromQL — fast 1-hour/5-minute and slow 6-hour/30-minute windows — with the complete alerting rules YAML, the matching Alertmanager route and receiver config, and a tuning method for eliminating alert fatigue while still catching genuine budget burn.

FAQ

Related

← Back to Monitoring & Observability for Read Replicas

• Alertmanager Rules for Replica Lag SLO Burn Rate — the multi-window multi-burn-rate PromQL and full rule YAML for staleness-SLO alerting referenced throughout this page.
• Prometheus Metrics for Replica Health and Lag — the exporter queries and metric definitions that these alert rules evaluate.
• Fallback Strategies When Replicas Fall Behind — what the read tier should do once a lag alert fires and a replica is pulled from the pool.
• Replication Lag & Consistency Management — the consistency model and SLO context that determines which lag thresholds actually matter for your workload.