Defining a Staleness SLO for Replica-Served Reads

Problem statement: you have a latency objective for reads and no objective at all for whether those reads returned current data — so the one property replicas actually put at risk is unmeasured.

A staleness SLO fixes that, and it uses exactly the machinery you already have for latency: a good-event ratio, an objective, an error budget, and burn-rate alerts. This page builds one. It depends on the served-age instrumentation described in tracing and SLO reporting for replica reads.

Symptom Identification

You need this objective if:

  • Your dashboards show replication lag per node and nothing about the freshness of what users received.
  • Lag alerts fire regularly, get acknowledged without action, and have been muted at least once.
  • A freshness incident was reported by a user rather than by monitoring.
  • The read path has a latency SLO at 99.9% and no equivalent statement about correctness of the data returned.
  • Nobody can answer “what fraction of reads last week were served from data older than five seconds”.

The muted lag alert is the clearest signal. It is muted because it fires on a condition that is usually harmless, which is precisely the failure of threshold alerting on an input rather than on an outcome.

Root Cause Analysis

Lag is an input; staleness is an outcome. Replication lag describes a node. Staleness describes what a user received. The mapping between them is not one-to-one: a lagging replica harms nobody if no read is routed to it, and a replica inside its lag budget can still serve a user a value older than they can tolerate if a cache sits in front of it.

Threshold alerts on inputs cannot be tuned to user impact. Set the lag threshold low and it fires constantly during batch windows when nobody is reading. Set it high and it misses a short lag spike during peak that affected thousands of requests. There is no threshold that is simultaneously quiet at 3 a.m. and sensitive at noon, because the missing variable is traffic, and a lag metric does not contain it.

A good-event ratio contains it automatically. Counting reads served within the bound over all reads weights every failure by how many requests it touched. A lag spike with no traffic contributes nothing; one during peak contributes proportionally. This is the same reason latency SLIs are expressed as ratios rather than as thresholds on p99.

The error budget is what makes the objective operationally useful. 99.5% over 30 days means roughly 216 minutes of complete failure, or a much longer period of partial failure. That number turns “should we buy another replica” from an argument into an arithmetic question: if lag events consumed 80% of the budget last month, the answer is yes.

Why a lag threshold and an error budget disagree Three stacked traces over one day. The first shows replication lag with two spikes of similar height, one at three in the morning and one at midday. The second shows read traffic, near zero overnight and high at midday. The third shows error budget consumption, which is negligible for the overnight spike and large for the midday one. A threshold alert on lag alone treats the two spikes identically; a burn-rate alert on the ratio does not. replication lag threshold 03:00 12:30 both breach — one alert cannot separate them read traffic ≈ 0 peak budget consumed < 1 % — nobody was reading 62 % of the month's budget in eleven minutes The burn-rate alert wakes someone for the second event and leaves them asleep for the first. That is the whole point.

Step-by-Step Resolution

Step 1 — Pick the bound from user impact

yaml
# slo/staleness.yml — one bound per read path, justified in a sentence.
# The justification is the useful part: it is what gets challenged in review.
read_paths:
  order_status:
    bound_s: 5
    why: "Users refresh after acting; a five-second-old status still matches
          what they just did. Above that they see the pre-action state and
          re-submit."
  product_listing:
    bound_s: 60
    why: "Stock counts change slowly relative to browsing; a minute-old count
          rarely changes a purchase decision."
  notification_count:
    bound_s: 120
    why: "A badge that lags two minutes is not noticed; a badge that is wrong
          when opened is."

Derive from behaviour, never from measurement. A bound set at the current p99 is met by definition and constrains nothing.

Inline verification: each why describes a user-visible consequence, not an infrastructure capability. If a justification mentions replication lag, it was derived backwards.

Step 2 — Express the SLI as a good-event ratio

promql
# Good events / all events, per read path. Structurally identical to a
# latency SLI so the same tooling, dashboards and review process apply.
sum by (db_read_path) (
  rate(db_read_served_age_ms_count{db_read_within_budget="true"}[5m])
)
/
sum by (db_read_path) (
  rate(db_read_served_age_ms_count[5m])
)

The within_budget label is computed at read time against that path’s own bound, which is what allows one metric to serve paths with different bounds.

Inline verification: the ratio is between 0 and 1 for every path and is defined (non-NaN) whenever traffic exists.

Step 3 — Set the objective and derive the budget

Objective Bad events allowed Full-outage equivalent per 30 days
99.0% 1 in 100 ~7 h 12 m
99.5% 1 in 200 ~3 h 36 m
99.9% 1 in 1000 ~43 m
99.95% 1 in 2000 ~21 m
promql
# Budget remaining, as a fraction, over the rolling 30-day window.
1 - (
  (1 - (sum(rate(db_read_served_age_ms_count{db_read_within_budget="true"}[30d]))
        / sum(rate(db_read_served_age_ms_count[30d]))))
  / 0.005                                    # 1 - objective, for 99.5 %
)

Choose the objective from what you can actually defend operationally. A 99.95% staleness objective on a read path fronted by an asynchronous replica in another region is a promise the architecture cannot keep, and an SLO you routinely miss is worse than none.

Inline verification: the remaining-budget expression yields a value near 1 in a healthy month and goes negative when the objective has been missed.

Step 4 — Alert on burn rate

yaml
groups:
  - name: staleness-slo
    rules:
      - record: read:staleness_good_ratio:5m
        expr: |
          sum by (db_read_path) (rate(db_read_served_age_ms_count{db_read_within_budget="true"}[5m]))
          / sum by (db_read_path) (rate(db_read_served_age_ms_count[5m]))
      - record: read:staleness_good_ratio:1h
        expr: |
          sum by (db_read_path) (rate(db_read_served_age_ms_count{db_read_within_budget="true"}[1h]))
          / sum by (db_read_path) (rate(db_read_served_age_ms_count[1h]))
      - record: read:staleness_good_ratio:6h
        expr: |
          sum by (db_read_path) (rate(db_read_served_age_ms_count{db_read_within_budget="true"}[6h]))
          / sum by (db_read_path) (rate(db_read_served_age_ms_count[6h]))

      # Fast burn: 14.4× exhausts a 30-day budget in ~2 days. Page.
      - alert: StalenessBudgetBurningFast
        expr: |
          (1 - read:staleness_good_ratio:5m) > 14.4 * 0.005
          and
          (1 - read:staleness_good_ratio:1h) > 14.4 * 0.005
        for: 2m
        labels: {severity: page}
        annotations:
          summary: "{{ $labels.db_read_path }} staleness budget burning 14x"
          runbook: "/replication-lag-consistency-management/fallback-strategies-when-replicas-fall-behind/"

      # Slow burn: 3× exhausts it in ~10 days. Ticket, do not page.
      - alert: StalenessBudgetBurningSlow
        expr: |
          (1 - read:staleness_good_ratio:6h) > 3 * 0.005
          and
          (1 - read:staleness_good_ratio:1h) > 3 * 0.005
        for: 30m
        labels: {severity: ticket}

The two-window requirement on each alert is what stops a single bad scrape paging: the short window makes it fast, the long window makes it true.

Inline verification: inject a two-minute freshness failure and confirm the fast-burn alert fires and then resolves; inject a low-level failure and confirm only the ticket fires.

Step 5 — Review the budget, not the incidents

promql
# The monthly review number: budget consumed per path.
1 - (
  sum by (db_read_path) (rate(db_read_served_age_ms_count{db_read_within_budget="true"}[30d]))
  / sum by (db_read_path) (rate(db_read_served_age_ms_count[30d]))
) / 0.005

A path that consumed 15% of its budget has headroom; one at 90% needs either investment or a renegotiated bound. Both are decisions the number makes straightforward.

Inline verification: the review produces a per-path percentage that the team can rank, rather than a list of incidents to discuss individually.

Configuration Snippet

yaml
# slo/staleness-complete.yml — objective, budget and alerting for one path,
# in the shape most SLO tooling expects.
slo:
  name: order-status-freshness
  service: checkout-api
  description: >
    Reads of order status are served from data no older than five seconds.

  sli:
    events:
      good:  db_read_served_age_ms_count{db_read_path="order_status",
                                         db_read_within_budget="true"}
      total: db_read_served_age_ms_count{db_read_path="order_status"}

  objective: 0.995            # 99.5 % → ~3 h 36 m of full-outage equivalent
  window: 30d

  alerting:
    page:
      burn_rate: 14.4
      windows: [5m, 1h]
    ticket:
      burn_rate: 3
      windows: [1h, 6h]

  # Excluded from the SLI, with a reason. Anything excluded silently is a
  # way of making the objective unfalsifiable.
  exclusions:
    - reason: "Planned failover windows, announced in advance"
      matcher: 'maintenance_window="true"'
    - reason: "Synthetic monitoring traffic — not user impact"
      matcher: 'db_read_path="order_status", client="synthetic"'

The exclusions block deserves scrutiny at every review. Each exclusion narrows what the objective promises, and exclusions accumulate quietly — a maintenance window here, a client there — until the SLI measures a small and unrepresentative slice of reality. Requiring a written reason per exclusion makes each one a decision rather than a default, and rereading them monthly is usually enough to catch the drift.

Error budget over a month, with the two burn-rate alert regimes Budget remaining declines from one hundred percent at the start of a thirty day window. A reference line shows the sustainable rate at which the budget lasts exactly the full window. A steep segment early in the month shows a fast burn that would exhaust the budget within two days and crosses the paging threshold. A shallower but persistent slope later shows a slow burn that would exhaust it in ten days and crosses the ticket threshold without ever paging. 100 % 0 % budget left day of the 30-day window → sustainable rate — lasts the window fast burn — 14.4× → page would exhaust the budget in ~2 days slow burn — 3× → ticket would exhaust it in ~10 days; nobody is woken The same SLI drives both alerts. Only the rate and the window differ, and the rate is what decides who is woken.

Verification and Rollback

Confirm the SLI reflects reality before trusting the alerts:

bash
# Force a controlled freshness failure and check the SLI responds.
for h in r1 r2 r3; do psql -h $h -c "SELECT pg_wal_replay_pause()"; done
sleep 120
for h in r1 r2 r3; do psql -h $h -c "SELECT pg_wal_replay_resume()"; done
promql
# During the pause this should visibly dip for any path with a tight bound.
read:staleness_good_ratio:5m

A ratio that does not move during a deliberate two-minute freshness failure means the instrumentation is not measuring what you think — most often because within_budget is computed against a bound so loose that a two-minute pause does not breach it.

Confirm the alerts have the right sensitivity:

bash
# Prometheus's own unit-test harness — cheaper than another live drill.
promtool test rules slo/staleness_test.yml

Rollback. Objectives and alerts revert independently, and the useful order is to relax rather than remove:

yaml
slo:
  objective: 0.990          # was 0.995 — relax while capacity is added
  alerting:
    page:   {burn_rate: 14.4, windows: [5m, 1h]}
    ticket: {burn_rate: 6,    windows: [1h, 6h]}   # was 3, quieter for now

If the objective proves unachievable, lowering it deliberately and recording why is far better than deleting it: the SLI keeps measuring, the trend stays visible, and the gap between the current objective and the desired one becomes the argument for investment. An SLO that is quietly removed takes the evidence with it.

Edge Cases and Gotchas

Low-traffic paths make the ratio noisy

A read path serving ten requests a minute produces a ratio that swings between 1.0 and 0.9 on a single bad read, which makes burn-rate alerting nearly useless. Either aggregate several low-traffic paths into one SLO, or add a minimum-events condition to the alert so it cannot fire on a handful of samples:

promql
and sum by (db_read_path) (rate(db_read_served_age_ms_count[5m])) > 1

The SLI must cover the whole path, including the client

If a browser or mobile client caches the response, the age the user perceives is the server-computed age plus the client’s cache duration, and the server-side histogram never sees it. Either subtract the client cache duration from the bound or propagate the computed age to the client so it can enforce the remainder — the mechanism in serving bounded-staleness reads with an explicit freshness budget.

A read that failed is not a fresh read

If a read errors — a timeout, a rejected escalation, an unavailable path — it produces no served-age observation and therefore silently vanishes from the denominator. That makes the staleness SLI improve during an outage, which is exactly backwards. Count failed reads as bad events explicitly rather than letting them fall out of the calculation.

FAQ

Why not just alert when replication lag exceeds a threshold?

Because lag is a node condition and the SLO is about user experience. A replica two seconds behind at three in the morning serving nobody consumes essentially no error budget; the same lag during peak consumes a great deal. A threshold alert treats those identically, which is why lag alerts get muted. A burn-rate alert on a staleness SLI fires in proportion to how many users were actually affected.

How do I pick the freshness bound?

Ask at what age the data becomes wrong rather than merely old, for the specific screen or API the read serves. A notification count can be a minute behind, an order status a few seconds, a balance shown before a transfer must be current. If you derive the bound from what the system currently achieves you get a number satisfied by construction that constrains nothing.

Should staleness and latency share one SLO?

Keep them separate, because they fail independently and have different remedies. A read can be fast and stale — served in three milliseconds from a replica forty seconds behind — or fresh and slow. Combining them into a single good-event definition is defensible when users genuinely require both at once, but it hides which one is failing, and that is the first question during an incident.

What if the SLO is met but users still complain?

The bound is wrong, or the SLI is measuring the wrong events. Check whether complaints concentrate on a read path with a looser bound than users expect, and whether the served-age instrumentation covers the whole path including client-side caching. An SLO met while users are unhappy is not a reason to distrust SLOs; it is a specific, actionable signal that the definition needs revising.

← Back to Tracing & SLO Reporting for Replica Reads