Tracing & SLO Reporting for Replica Reads

Most read-replica monitoring answers questions about nodes: how far behind is replica three, is its pool saturated, how much CPU is it burning. Those are necessary and they are not sufficient, because they cannot answer the question anyone actually asks during an incident — which user requests were affected, and how. A replica can be two seconds behind and harm nobody, or two hundred milliseconds behind and break a checkout flow, and node-level dashboards look identical in both cases.

This page closes that gap. It is about attaching the routing decision to the request that caused it, so that replica monitoring and observability can express replica behaviour as an objective about user-visible reads rather than an inventory of node states.

Concept and Scope: From Node Health to Read Quality

Three levels of observability exist for a read tier, and they answer different questions.

Node level. Per-replica lag, connections, CPU, cache hit ratio. Answers “is this node healthy”. Built from exporters, covered in Prometheus metrics for replica health and lag.

Decision level. For each query, which node was chosen and why. Answers “is the routing working”. This is where tracing earns its place, because the reason for a routing decision exists only inside the router at the moment of the decision and is unrecoverable afterwards.

Outcome level. For each user-facing read, was it fast enough and fresh enough. Answers “are users getting what we promised”. This is the SLO layer, and it is what a weekly report should be about.

The three compose upward. Node metrics feed decisions; decisions determine outcomes. The common failure is to build the first level thoroughly, skip the second, and attempt the third by inference — producing dashboards that show healthy replicas next to an unexplained rise in primary load.

Two SLIs, not one. A read has two quality dimensions, and only one of them is conventionally measured. Latency — did it return quickly. Freshness — was the data recent enough. A read served in 3 ms from a replica 40 seconds behind scores perfectly on a latency SLI and may be entirely useless. Any serious read-tier objective needs both, and the freshness one is nearly always the missing half.

Scope: this page is about instrumenting and reporting. Choosing the freshness bound for a given read path is a design decision covered in routing queries based on data freshness requirements; what follows assumes a bound exists and concerns itself with measuring compliance.

Three levels of read-tier observability and the question each answers Three stacked bands. The bottom band is node level: lag, connections and CPU per replica, answering whether a node is healthy. The middle band is decision level: per-query span attributes recording the node chosen and the reason, answering whether routing is working. The top band is outcome level: latency and freshness indicators against a budget, answering whether users got what was promised. Arrows run upward, and a note observes that skipping the middle band forces the top to be inferred rather than measured. outcome — SLIs latency within bound · freshness within bound · budget consumed “did users get what we promised?” decision — span attributes node chosen · reason · lag at selection · fallback taken “is routing working?” node — exporter metrics replication lag · pool saturation · CPU · buffer hit ratio “is this node healthy?” skip the middle band and the top one must be guessed

Mechanism Deep-Dive: Attributes That Make a Trace Diagnostic

A database span that records only duration tells you a query was slow. A database span that records the routing decision tells you why — and the difference is entirely in a handful of attributes set at the moment of selection.

python
# Instrument at the router, where the decision is actually made. Setting these
# after the fact from a connection object loses `reason`, which is the whole point.
from opentelemetry import trace

tracer = trace.get_tracer("db.router")

def execute_read(sql, params, *, freshness_budget_s):
    with tracer.start_as_current_span("db.query") as span:
        choice = router.select(freshness_budget_s)   # returns node, reason, lag

        # Standard semantic-convention attributes — keep these conventional so
        # existing tooling groups them correctly.
        span.set_attribute("db.system", "postgresql")
        span.set_attribute("server.address", choice.node.host)
        span.set_attribute("db.namespace", choice.node.dbname)

        # Routing attributes — the ones that make this span diagnostic.
        span.set_attribute("db.replica.role", choice.node.role)          # primary | replica
        span.set_attribute("db.replica.instance", choice.node.name)      # r1, r2, …
        span.set_attribute("db.replica.select_reason", choice.reason)
        #   eligible | fallback_lag | fallback_no_healthy | pinned_txn
        #   | pinned_watermark | forced_primary
        span.set_attribute("db.replica.lag_ms", choice.lag_ms)
        span.set_attribute("db.read.freshness_budget_ms", freshness_budget_s * 1000)

        rows = choice.node.execute(sql, params)

        # Outcome: the age of what was actually served, not the lag at selection.
        span.set_attribute("db.read.served_age_ms", choice.observed_age_ms())
        span.set_attribute("db.read.within_budget",
                           choice.observed_age_ms() <= freshness_budget_s * 1000)
        return rows

db.replica.select_reason is the highest-value attribute in the list and the one most often missing. Consider two traces that both show a read served by the primary. One has reason=fallback_lag — the replicas were behind, the freshness gate did its job, and the system behaved correctly under stress. The other has reason=pinned_txn — an ORM opened a transaction eagerly and pinned an ordinary read to the primary, which is a bug costing replica capacity on every request. Without the attribute the two are indistinguishable, and the second one is invisible forever.

Emit metrics from the same spans

Deriving dashboard metrics from span attributes rather than from a separate code path guarantees the trace view and the dashboard cannot contradict each other during an incident — a failure mode that wastes more time than almost any other.

python
# One span, two outputs: a sampled trace and an unsampled metric.
read_age = meter.create_histogram(
    "db.read.served_age", unit="ms",
    description="Age of the data a read actually returned")

read_age.record(choice.observed_age_ms(), {
    "path": route_name,                        # the user-facing read path
    "role": choice.node.role,
    "reason": choice.reason,
    "within_budget": str(within_budget).lower(),
})

Because the histogram is recorded on every span while only a fraction of traces are exported, the SLI is computed from complete data and the traces remain exemplars. This is the standard resolution to the cost objection: sample the expensive artefact, keep the cheap aggregate exact.

Attribute latency to the selection, not just to the query

A read’s total latency has parts that node metrics cannot separate: time spent waiting for a pool checkout, time spent evaluating the freshness gate, and time spent executing. Child spans make the split visible.

python
with tracer.start_as_current_span("db.query") as span:
    with tracer.start_as_current_span("db.route.select"):
        choice = router.select(freshness_budget_s)       # gate + selection
    with tracer.start_as_current_span("db.pool.checkout"):
        conn = choice.node.checkout()                     # queueing shows up here
    with tracer.start_as_current_span("db.execute"):
        rows = conn.execute(sql, params)

When p99 read latency rises, this decomposition immediately distinguishes a slow database from a saturated pool from a freshness gate rejecting replicas and falling back across a region — three problems with three different remediations that look identical in an aggregate latency graph.

Trade-off Comparison: Signal Sources

Source Answers Cardinality cost Retention Blind to
Exporter metrics (node) Is a node healthy Low — per node Long, cheap Which requests were affected
Span attributes (decision) Why this query went where it did Moderate — per attribute value Short, sampled Nothing about aggregate volume unless metrics are derived
Metrics derived from spans What fraction of reads met the bound Moderate — keep label sets small Long Individual request context
Query digests (pg_stat_statements) Which statements are expensive Low Medium Routing decisions entirely
Application logs Arbitrary detail Very high at volume Short Aggregation without heavy processing

The important line is the second: span attributes are per-decision, which makes them the only source that can explain a routing outcome, and also the one whose cardinality gets out of hand fastest. Keep select_reason to a small closed enumeration and never put a user identifier or a raw SQL string in a metric label. Traces can carry high-cardinality context; metrics derived from them must not.

Configuration Runbook

Recording rules for the two SLIs

yaml
# prometheus/rules/read_slo.yml
groups:
  - name: replica-read-slo
    interval: 30s
    rules:
      # Freshness SLI: fraction of reads served within their stated budget.
      - record: read:freshness_good_ratio:5m
        expr: |
          sum by (path) (rate(db_read_served_age_ms_count{within_budget="true"}[5m]))
          / sum by (path) (rate(db_read_served_age_ms_count[5m]))

      # Latency SLI: fraction of reads completing inside the latency objective.
      - record: read:latency_good_ratio:5m
        expr: |
          sum by (path) (rate(db_query_duration_ms_bucket{le="200"}[5m]))
          / sum by (path) (rate(db_query_duration_ms_count[5m]))

      # Fallback rate, split by cause — the diagnostic companion to both.
      - record: read:fallback_ratio_by_reason:5m
        expr: |
          sum by (path, reason) (rate(db_read_served_age_ms_count{role="primary"}[5m]))
          / sum by (path) (rate(db_read_served_age_ms_count[5m]))

Keeping the two SLIs structurally identical is deliberate: the same burn-rate machinery, dashboard panels and review process then applies to both, and freshness stops being a second-class concern maintained by whoever remembers it.

Multi-window burn-rate alerts

yaml
# Page on fast burn; ticket on slow burn. A 99.5 % objective over 30 days
# means a 14.4× burn rate exhausts the month's budget in about two days.
groups:
  - name: replica-read-burn
    rules:
      - alert: ReadFreshnessBudgetBurningFast
        expr: |
          (1 - read:freshness_good_ratio:5m)  > 14.4 * 0.005
          and
          (1 - read:freshness_good_ratio:1h)  > 14.4 * 0.005
        for: 2m
        labels: {severity: page}
        annotations:
          summary: "Read freshness budget burning 14x on {{ $labels.path }}"
          runbook: "/replication-lag-consistency-management/fallback-strategies-when-replicas-fall-behind/"

      - alert: ReadFreshnessBudgetBurningSlow
        expr: |
          (1 - read:freshness_good_ratio:6h)  > 3 * 0.005
          and
          (1 - read:freshness_good_ratio:24h) > 3 * 0.005
        for: 30m
        labels: {severity: ticket}

The two-window requirement on each alert is what prevents a single scrape blip from paging. The short window makes the alert fast; the long window makes it true. This is the same construction used for lag SLO burn in Alertmanager rules for replica lag SLO burn, applied to the freshness objective instead of the node metric.

Tail-based sampling that keeps the interesting traces

yaml
# otel-collector: sample cheaply, but never drop a trace you would want to read.
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-fallbacks              # every read that missed its budget
        type: string_attribute
        string_attribute:
          key: db.replica.select_reason
          values: [fallback_lag, fallback_no_healthy, pinned_watermark]
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 1}

Head-based sampling at 1% throws away 99% of the fallback traces, which are the entire reason the instrumentation exists. Tail-based sampling keeps all of them and 1% of the boring ones, for roughly the same export volume.

Two traced reads: replica-served versus freshness fallback Two waterfall traces on a common millisecond axis. The first has a short route-selection span, a short pool checkout, and a short execute on a replica, totalling about twelve milliseconds. The second has a long route-selection span because the freshness gate rejected each replica in turn, followed by a checkout and execute against the primary, totalling about ninety milliseconds. The select reason attribute distinguishes them, and without it both would appear only as a slow query. 0 ms 40 ms 80 ms 120 ms served by replica reason = eligible ≈ 12 ms · lag at selection 40 ms · within budget fell back to primary reason = fallback_lag gate rejects r1, r2, r3 execute on primary ≈ 90 ms · correct behaviour, expensive route select pool checkout execute Without the decomposition both traces are just “a database call”; with it, the second one names its own cause. A rising share of the second shape is the earliest visible sign that the read tier is losing headroom.

Monitoring and Alerting Signals

promql
# 1. Freshness SLI per read path — the headline number for the weekly report.
read:freshness_good_ratio:5m

# 2. Fallback share by cause. A rise in fallback_lag is capacity;
#    a rise in pinned_txn is a code regression. Very different responses.
topk(5, read:fallback_ratio_by_reason:5m)

# 3. Selection overhead — how long the gate itself takes. If this grows,
#    the router is probing too many nodes or the lag source is slow.
histogram_quantile(0.99, sum by (le) (rate(db_route_select_duration_ms_bucket[5m])))

# 4. Share of reads served by each replica, from spans rather than from the proxy.
sum by (instance) (rate(db_read_served_age_ms_count{role="replica"}[5m]))
  / ignoring(instance) group_left sum (rate(db_read_served_age_ms_count{role="replica"}[5m]))

# 5. Traces retained per fallback reason — confirms the sampler is keeping
#    what you will actually want to read during the next incident.
sum by (reason) (rate(otelcol_processor_tail_sampling_sampled_total[30m]))

Signal 2 deserves a dedicated dashboard panel. The distinction between fallback because the data was not fresh enough and fallback because the code pinned the session is the single most actionable split in the whole read tier, and it exists nowhere except in the select_reason attribute. Signal 4 is worth cross-checking against the proxy’s own view: a disagreement between what the router believes it distributed and what the application observed being served is a strong signal of connection-level pinning, discussed in load balancing reads across a replica pool.

Failure Modes and Recovery Steps

  1. Traces show the node but not the reason. Diagnosis: you can see reads hitting the primary and cannot tell whether that is the freshness gate working or a bug. Recovery: none available after the fact — the reason is not recoverable from the data. Prevention: set select_reason at the point of decision, before the query runs.

  2. SLI and dashboard disagree during an incident. Diagnosis: the trace view and the Grafana panel tell different stories, and the incident stalls while people argue about which is right. Recovery: trust neither until reconciled. Prevention: derive the metric from the span, so there is one measurement with two renderings.

  3. Metric cardinality explosion after adding attributes. Diagnosis: Prometheus memory climbs sharply after an instrumentation deploy; series count per metric is in the hundreds of thousands. Recovery: drop the offending label with a metric_relabel_config immediately, then fix the emitter. Prevention: keep metric labels to a closed enumeration — path, role, reason, within_budget — and let traces carry everything of higher cardinality.

  4. Head-based sampling discards every interesting trace. Diagnosis: the fallback rate metric is clearly non-zero but no fallback trace can be found. Recovery: switch to tail-based sampling with an explicit keep policy on fallback reasons. Prevention: signal 5, so the sampler’s own behaviour is observable.

  5. A freshness SLO that nobody set a bound for. Diagnosis: within_budget is true for essentially every read because the budget was set generously enough to be unfalsifiable. Recovery: derive the bound from the read path’s actual requirement, not from what the current system achieves. Prevention: set the bound during design, before the instrumentation exists, so it is a requirement rather than a description.

Production-Readiness Checklist

In This Section

FAQ

What span attributes should a replica read carry?

At minimum: the server address and instance name of the node that served it, whether that node was a primary or a replica, the reason the router chose it, and the replication lag observed at selection time. The reason attribute is the one teams omit and the one that makes traces diagnostic rather than merely descriptive — without it you can see that a read went to the primary but not whether that was a freshness gate, a transaction pin, or a classification bug.

How do I turn staleness into an SLI?

Define it as the proportion of reads served within a stated freshness bound, exactly parallel to a latency SLI. If your bound is two seconds and 99.5% of reads are served from data no older than that, your staleness SLI is 99.5%. This requires the read path to know and emit the age of what it served, which is a small instrumentation cost and the only way freshness becomes a first-class objective rather than an assumption.

Why alert on burn rate instead of on lag crossing a threshold?

Because a lag threshold fires on a condition, not on a consequence. A replica two seconds behind at three in the morning with no traffic consumes almost no error budget; the same lag during peak consumes a great deal. Burn-rate alerting is proportional to user impact, which means it wakes people for the second case and not the first, and it is what keeps a lag alert from becoming background noise everyone has learned to ignore.

Does tracing every database call cost too much?

Tracing every call and exporting every trace does. The standard resolution is to sample traces for export while emitting metrics from 100% of spans, so aggregate SLI numbers are exact and only the individual exemplars are sampled. Keep the sampler tail-based where possible so that slow and fallback reads are retained preferentially — those are the traces you will actually open.

← Back to Monitoring & Observability for Read Replicas