Alertmanager Rules for Replica Lag SLO Burn Rate

Problem statement: Your replica-lag alert either pages too often on transient spikes or too late on a slow drift that quietly erodes read freshness for a whole day — and you need an SLO-based rule set that pages fast on acute burn, tickets on chronic burn, and stays quiet the rest of the time.

Symptom Identification

You need burn-rate alerting when your current lag alerting shows one or more of these patterns:

• The pager fires several times a day on ReplicaLagHigh, each lasting under a minute, and on-call has stopped reading it.
• A replica sat at 3-4 seconds of lag — just under your hard threshold — for six hours, degrading read-your-writes correctness the whole time, and nothing alerted because no single scrape crossed the ceiling long enough to matter.
• After an incident resolves, the lag alert keeps firing for another 20-30 minutes because it is averaged over a long window with no fast reset.
• Leadership asks “what fraction of the month did reads meet the freshness target?” and you have no number to give them because your alerts are threshold events, not budget measurements.

These are all symptoms of threshold-based alerting without an error budget. A burn-rate SLO alert replaces the binary “over/under the line” with “how fast are we spending our allowance of stale reads.”

Root Cause Analysis

A raw threshold alert answers the wrong question. It asks is lag over X right now? The question that maps to user pain is are we consuming our staleness budget faster than the month can absorb? Those diverge in two directions:

1. Brief spikes over the threshold are cheap. A replica at 5 seconds of lag for 90 seconds consumes a negligible slice of a 30-day budget. Paging on it is pure noise. The threshold alert cannot tell the difference between 90 seconds and 9 hours of breach — both look like “over the line.”

2. Sustained sub-threshold degradation is expensive. A replica parked at 3.5 seconds when your ceiling is 4 seconds never trips the threshold, yet if your SLO ceiling for good is 2 seconds, every one of those scrapes is a budget-consuming bad event. Over six hours that quietly burns a large fraction of the month’s allowance with zero alerts.

The fix is to define the SLO as a ratio of good scrapes, derive an error budget from it, and alert on the burn rate — the multiple of normal budget consumption. The parent guide on alerting for replication lag and pool saturation covers the absolute-threshold and rate-of-change rules that complement this; the burn-rate rules here are the budget-aware layer on top. All of them read from the metrics defined in the Prometheus metrics for replica health and lag guide.

Architecture

The burn-rate design runs two alerts, each combining a long and a short window. The long window confirms the burn is sustained; the short window lets the alert clear fast once it stops.

Step-by-Step Resolution

Step 1: Define the staleness SLO

Pick a lag ceiling that marks the boundary between a fresh and a stale read for your workload, and a compliance target over a rolling window. A reasonable user-facing default:

SLO: 99.5% of replica-lag scrapes are below 2s, measured over a rolling 30 days.
  - Good event  = a scrape where pg_replication_lag_seconds < 2
  - Bad event   = a scrape where pg_replication_lag_seconds >= 2
  - Compliance target = 99.5%  ->  error budget = 0.5% of scrapes may be bad

The ceiling (2s) should match the freshness requirement of the reads you route to replicas — tighten it for read-your-writes-sensitive tiers, loosen it for analytics.

Inline verification: count_over_time(pg_replication_lag_seconds[30d]) should return a non-zero sample count per replica; if it is empty, the metric is not being retained for the SLO window and the budget cannot be computed.

Step 2: Compute the error budget and burn rates

The error budget is the allowed fraction of bad scrapes: 1 - 0.995 = 0.005 (0.5%). Burn rate is how many times faster than “budget-neutral” you are consuming it. A burn rate of 1 exactly exhausts the budget at the end of the window; higher multiples exhaust it sooner:

Budget = 0.5% bad scrapes over 30 days.
Burn rate = (observed bad ratio) / (budget fraction 0.005)

  14.4x burn -> 30d budget gone in ~2 days   -> PAGE  (fast burn)
   6x  burn -> 30d budget gone in ~5 days    -> TICKET (slow burn)
   1x  burn -> budget lasts exactly 30 days  -> no alert

Inline verification: compute the current burn instantly with (rate over 1h of bad scrapes) / 0.005. A value near or below 1 means you are within budget; a value of 14.4 or higher over a sustained hour is a fast-burn condition.

Step 3: Add recording rules for the bad-scrape ratio

Pre-compute the bad ratio per window so the alert expressions stay cheap and readable. A scrape is bad when lag is at or above the ceiling:

groups:
  - name: replica_staleness_slo_recording
    interval: 30s
    rules:
      - record: replica:lag_bad_ratio:5m
        expr: |
          sum by (cluster) (rate(pg_replication_lag_bad_scrapes_total[5m]))
          / sum by (cluster) (rate(pg_replication_lag_total_scrapes_total[5m]))
      - record: replica:lag_bad_ratio:1h
        expr: |
          sum by (cluster) (rate(pg_replication_lag_bad_scrapes_total[1h]))
          / sum by (cluster) (rate(pg_replication_lag_total_scrapes_total[1h]))
      - record: replica:lag_bad_ratio:30m
        expr: |
          sum by (cluster) (rate(pg_replication_lag_bad_scrapes_total[30m]))
          / sum by (cluster) (rate(pg_replication_lag_total_scrapes_total[30m]))
      - record: replica:lag_bad_ratio:6h
        expr: |
          sum by (cluster) (rate(pg_replication_lag_bad_scrapes_total[6h]))
          / sum by (cluster) (rate(pg_replication_lag_total_scrapes_total[6h]))

If you do not have dedicated _bad_scrapes_total counters, derive the ratio directly from the lag gauge instead:

      - record: replica:lag_bad_ratio:1h
        expr: |
          avg_over_time(
            (pg_replication_lag_seconds{role="replica"} >= bool 2)[1h:30s]
          )

The >= bool 2 produces 1 for bad scrapes and 0 for good ones; averaging over the window gives the bad-scrape fraction directly.

Inline verification: replica:lag_bad_ratio:1h should sit near 0 in steady state. A value of 0.072 means 7.2% of the last hour’s scrapes were stale — a burn rate of 0.072 / 0.005 = 14.4x.

Step 4: Write the multi-window burn-rate alerts

Each alert ANDs a long window (confirms sustained burn) with a short window (fast reset guard):

groups:
  - name: replica_staleness_slo_alerts
    rules:
      - alert: ReplicaStalenessFastBurn
        expr: |
          replica:lag_bad_ratio:1h > (14.4 * 0.005)
          and
          replica:lag_bad_ratio:5m > (14.4 * 0.005)
        for: 2m
        labels:
          severity: critical
          slo: replica_staleness
        annotations:
          summary: "Cluster {{ $labels.cluster }} burning staleness budget at >14.4x"
          runbook_url: "https://runbooks.internal/db/staleness-fast-burn"
      - alert: ReplicaStalenessSlowBurn
        expr: |
          replica:lag_bad_ratio:6h > (6 * 0.005)
          and
          replica:lag_bad_ratio:30m > (6 * 0.005)
        for: 15m
        labels:
          severity: warning
          slo: replica_staleness
        annotations:
          summary: "Cluster {{ $labels.cluster }} burning staleness budget at >6x"
          runbook_url: "https://runbooks.internal/db/staleness-slow-burn"

Inline verification: in the Prometheus expression browser, evaluate the fast-burn expression; it must return empty in steady state. Force it by temporarily lowering the multiplier to 0 — it should then return one series per cluster, confirming the label wiring.

Step 5: Route fast and slow burn to different receivers

Fast burn pages; slow burn opens a ticket. Grouping by cluster and slo keeps a multi-replica cluster to one notification:

route:
  group_by: ['cluster', 'slo']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: chat-info
  routes:
    - matchers: [slo="replica_staleness", severity="critical"]
      receiver: pager-critical
      group_wait: 10s
    - matchers: [slo="replica_staleness", severity="warning"]
      receiver: ticket-warning

receivers:
  - name: pager-critical
    pagerduty_configs:
      - routing_key: "<pd-routing-key>"
  - name: ticket-warning
    webhook_configs:
      - url: "http://ticketer.internal/staleness-burn"
  - name: chat-info
    slack_configs:
      - channel: "#db-slo"
        api_url: "<slack-webhook-url>"

Inline verification: send a synthetic alert with amtool alert add ReplicaStalenessFastBurn slo=replica_staleness severity=critical cluster=test and confirm it lands in PagerDuty, not Slack.

Configuration Snippet

A single self-contained rule file combining recording and alerting rules for a 99.5% / 2s / 30-day SLO:

groups:
  - name: replica_staleness_slo
    interval: 30s
    rules:
      - record: replica:lag_bad_ratio:5m
        expr: avg_over_time((pg_replication_lag_seconds{role="replica"} >= bool 2)[5m:30s])
      - record: replica:lag_bad_ratio:1h
        expr: avg_over_time((pg_replication_lag_seconds{role="replica"} >= bool 2)[1h:30s])
      - record: replica:lag_bad_ratio:30m
        expr: avg_over_time((pg_replication_lag_seconds{role="replica"} >= bool 2)[30m:30s])
      - record: replica:lag_bad_ratio:6h
        expr: avg_over_time((pg_replication_lag_seconds{role="replica"} >= bool 2)[6h:30s])
      - alert: ReplicaStalenessFastBurn
        expr: replica:lag_bad_ratio:1h > 0.072 and replica:lag_bad_ratio:5m > 0.072
        for: 2m
        labels: {severity: critical, slo: replica_staleness}
        annotations:
          runbook_url: "https://runbooks.internal/db/staleness-fast-burn"
      - alert: ReplicaStalenessSlowBurn
        expr: replica:lag_bad_ratio:6h > 0.03 and replica:lag_bad_ratio:30m > 0.03
        for: 15m
        labels: {severity: warning, slo: replica_staleness}
        annotations:
          runbook_url: "https://runbooks.internal/db/staleness-slow-burn"

The literals 0.072 and 0.03 are 14.4 * 0.005 and 6 * 0.005 pre-multiplied — keep them in sync with the SLO if you change the compliance target.

Verification and Rollback

Confirm the rules load and evaluate correctly:

# Validate rule syntax before reloading Prometheus
promtool check rules /etc/prometheus/rules/replica_staleness_slo.yml

# Unit-test the burn logic with sample series
promtool test rules /etc/prometheus/tests/staleness_slo_test.yml

# Reload without restarting
curl -X POST http://localhost:9090/-/reload

Confirm the alerts behave against real data by querying the recording rules and checking that steady-state burn sits well below the thresholds:

replica:lag_bad_ratio:1h        # expect near 0 in steady state
ALERTS{slo="replica_staleness"} # expect empty unless genuinely burning

To roll back if the new rules are noisy or wrong, restore the previous file and reload:

cd /etc/prometheus/rules/
git checkout HEAD~1 -- replica_staleness_slo.yml
curl -X POST http://localhost:9090/-/reload

Because burn-rate rules are additive, you can also silence just the SLO alerts in Alertmanager while you tune, without touching the threshold alerts from the parent alerting guide: amtool silence add slo=replica_staleness --duration=2h --comment="tuning burn thresholds".

Edge Cases and Gotchas

Subquery cost at scale

The [6h:30s] subquery in the derived-ratio form re-evaluates the inner expression across a large range on every rule evaluation. On a fleet of hundreds of replicas this is expensive and can slow rule-group evaluation past its interval, delaying every alert. Prefer real counters (..._bad_scrapes_total incremented by the exporter) over subqueries once you outgrow a handful of replicas, and keep interval: 30s on the recording group so the ratios are pre-materialized and the alert expressions become cheap gauge comparisons.

Low traffic makes the ratio jumpy

Burn rate is a ratio, so at low scrape counts a single bad scrape can swing the short-window ratio dramatically — one bad scrape in a 5-minute window at 30s scrape interval is 1/10 = 10%, instantly over the fast-burn line. The long-window half of the AND is what protects you: the 1-hour window needs roughly 72 bad scrapes to reach 7.2%, which a single blip cannot produce. Never alert on the short window alone. If a replica fleet genuinely has too few scrapes to compute a stable ratio, lengthen the evaluation window or fall back to the absolute-threshold rule instead.

Budget resets and burn-alert amnesia

A burn-rate alert measures the rate of consumption, not the remaining budget. It is entirely possible to have exhausted the whole 30-day budget in the first week yet have every burn alert quiet because current burn dropped back to 1x. Pair the burn alerts with a slower budget-remaining panel or alert (1 - (30d bad ratio / 0.005)) so a fully-spent budget is visible even when instantaneous burn looks healthy. When the budget is spent, the correct operational response is to stop routing freshness-sensitive reads to replicas — see fallback strategies when replicas fall behind.

FAQ

Related

← Back to Alerting on Replication Lag and Pool Saturation

• Prometheus Metrics for Replica Health and Lag — the lag metrics and exporter counters these recording rules aggregate into a bad-scrape ratio.
• Fallback Strategies When Replicas Fall Behind — what to do operationally once the staleness budget is burning or spent.
• Monitoring & Observability for Read Replicas — the broader observability practice this SLO alerting slots into.