Building a Replica Lag Heatmap in Grafana

Problem statement: You run ten or more read replicas, and when one falls behind on replication your time-series lag panel becomes an unreadable tangle of overlapping lines — you can see something is lagging but not which node, and a fleet average hides the outlier entirely. A heatmap collapses the whole fleet onto one panel where a single lagging replica shows up as one hot band.

Symptom Identification

You need a heatmap (rather than another line chart) when you see these signals on your existing dashboard:

  • A lag time-series panel with fifteen overlapping series where no individual line is distinguishable — the classic “spaghetti graph.”
  • A fleet-summary avg(pg_replication_lag_seconds) that stays under SLO while individual users report stale reads, because one replica at 8 s is diluted by nine replicas at 100 ms.
  • Alerts firing for “a replica” over its lag SLA, but the on-call engineer has to run an ad-hoc topk() query to find which instance, costing minutes during an incident.
  • Intermittent lag spikes that you suspect correlate across replicas (shared storage, shared network) but cannot confirm because per-node lines are visually separated.

A heatmap addresses all four: every replica occupies its own row (state-timeline) or contributes to a density band (Heatmap), so one node deviating from the fleet is immediately visible, and correlated spikes appear as a vertical column of hot cells.

Root Cause Analysis

The reason a line chart fails at fleet scale is that it encodes value on the Y axis and uses color only to separate series. With fifteen replicas you have fifteen colors overlapping in the same vertical space; the eye cannot track any one of them, and the important signal — one line diverging from the pack — is exactly the case where lines cross and occlude each other most.

A heatmap re-encodes the data. Time stays on the X axis, but the replica identity (or a lag bucket) moves to the Y axis and the lag magnitude becomes color intensity. Now each replica has a dedicated horizontal lane that never occludes another. A lagging node is a hot lane against cool neighbors — a pre-attentive visual pattern the eye catches without reading a single number. This is the same distribution-over-time telemetry described in detecting and handling replication lag in real-time, rendered for fleet-scale comparison rather than single-node depth.

The choice between Grafana’s Heatmap panel and its state-timeline panel comes down to what you are optimizing for. The Heatmap panel buckets values and shows density — best when you want the continuous distribution of lag across many nodes. The state-timeline gives one labelled row per replica with discrete SLA-band colors — best when the fleet is bounded and you want to name the offending node instantly. For most replica fleets under thirty nodes, the state-timeline is the more legible default, and this guide builds that first, then covers the true Heatmap panel.

Architecture

The panel maps three dimensions — time, replica, and lag — onto a two-dimensional grid with color as the third channel. The diagram shows the mapping and how the SLA thresholds become color bands, with one replica lagging.

Replica lag heatmap layout with one lagging node A grid with four replica rows on the Y axis and eight time columns on the X axis. Three rows are cool (within SLA), one row is hot across recent columns showing a replica over its lag budget. A legend on the right maps three color bands to under 0.5 seconds, 0.5 to 5 seconds warning, and over 5 seconds critical. lag heatmap replica (Y) time → (X) replica-a replica-b replica-c replica-d one hot lane = one lagging replica SLA bands < 0.5s ok 0.5–5s warn > 5s critical

Step-by-Step Resolution

Step 1: Write the per-replica lag query

The query must return one series per replica, sampled over time. Query the lag metric directly — do not aggregate across instances, because the whole point is to keep each replica separate. Set a Min step on the query so the panel does not request finer resolution than Prometheus scrapes.

text
# Panel query A — one series per replica, native lag in seconds
pg_replication_lag_seconds{job="postgres_replicas"}

# In the query options set:
#   Min step: 15s        (>= your Prometheus scrape interval)
#   Format:   Time series
#   Legend:   {{instance}}

Inline verification: In Grafana’s query inspector, the response should contain one frame per replica instance, each with an instance label. If you see a single merged frame, an aggregation crept into the query — remove any sum/avg wrapping.

Step 2: Choose and add the panel type

For a named-row view, add a State timeline panel; for a density view, add a Heatmap panel. Start with the state-timeline — it labels each replica and is the fastest to read during an incident.

text
Panel type:  State timeline
Merge:       off
Value:       Last (non-null)
Show values: never   (color carries the signal, not text)

Inline verification: The panel should render one horizontal lane per replica, each labelled with its instance. If lanes are missing, the template variable or job label is filtering them out — widen the query selector.

Step 3: Configure buckets (Heatmap) or bands (state-timeline)

For the state-timeline, lag is continuous, so map it into discrete SLA bands using value mappings or thresholds (Step 4 handles color). For a true Heatmap panel, set it to compute buckets from the incoming values rather than expecting pre-bucketed data.

text
# Heatmap panel bucketing (Panel > Heatmap options)
Calculate from data: Yes
Y axis:              Linear, unit = seconds (s)
Y bucket:            Size 1        # 1-second lag buckets; align to SLA granularity
X bucket:            $__interval   # let Grafana match the dashboard time range

Inline verification: The Y axis should show lag in seconds, not a raw sample count. If it shows counts, the panel is treating your data as pre-bucketed histogram series — switch “Calculate from data” on.

Step 4: Map colors to the lag SLA

This is the step that makes the panel actionable. Set the color scheme to Thresholds mode and place the steps at your SLO warning and critical values so a hot cell means an actual breach, using the same numbers as your alerting on replication lag and pool saturation rules.

json
{
  "color": {"mode": "thresholds"},
  "thresholds": {
    "mode": "absolute",
    "steps": [
      {"color": "green",  "value": null},
      {"color": "yellow", "value": 0.5},
      {"color": "red",    "value": 5}
    ]
  }
}

Inline verification: Temporarily lower the critical step to a value below a healthy replica’s current lag; that replica’s lane must turn red. Restore the real threshold afterward.

Step 5: Read the heatmap to find the lagging replica

With colors bound to the SLA, reading is pre-attentive. Scan for a single hot lane against cool neighbors — that is one replica over budget (a bad node, not a fleet event). Scan for a vertical column of hot cells across all lanes — that is a correlated event (shared storage, network, a bulk load on the primary). A lane that flickers between warm and cool is a flapping replica worth investigating for scrape or GC noise.

text
Interpretation guide:
  one hot lane, cool neighbours   -> single lagging replica; eject or investigate that node
  vertical hot column, all lanes  -> fleet-wide cause; check primary / shared infra
  intermittent warm on one lane   -> flapping; check exporter latency, GC pauses

Inline verification: Cross-check the hot lane’s instance against topk(1, pg_replication_lag_seconds) in Explore — the instance names must match.

Configuration Snippet

A minimal state-timeline panel JSON for a fleet lag view, ready to paste into a provisioned dashboard’s panels array:

json
{
  "type": "state-timeline",
  "title": "Fleet replication lag vs SLA",
  "datasource": {"uid": "prom-main"},
  "targets": [
    {
      "expr": "pg_replication_lag_seconds{job=\"postgres_replicas\"}",
      "legendFormat": "{{instance}}",
      "interval": "15s"
    }
  ],
  "options": {"mergeValues": false, "showValue": "never", "rowHeight": 0.9},
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green",  "value": null},
          {"color": "yellow", "value": 0.5},
          {"color": "red",    "value": 5}
        ]
      }
    }
  }
}

Verification and Rollback

Confirm the panel is telling the truth before you rely on it during an incident:

text
# 1. Induce lag on a test replica (staging only) and watch the lane heat up
#    e.g. pause WAL replay or run a heavy read that blocks apply
# 2. Confirm the hot instance matches an independent query:
topk(3, pg_replication_lag_seconds{job="postgres_replicas"})
# 3. Confirm the red band aligns with the alert rule firing for the same instance

If the panel renders incorrectly — wrong axis unit, merged series, missing lanes — roll back by restoring the previous dashboard JSON from version control rather than editing in the UI:

bash
git checkout HEAD~1 -- dashboards/replica-fleet.json
# Grafana reloads provisioned dashboards on its updateInterval; force it if needed:
systemctl reload grafana-server

Because the dashboard is provisioned as code, the rollback is a single revert with no risk of leaving a half-edited panel live. The provisioning setup itself is covered in the parent guide, Grafana dashboards for read replica fleets.

Edge Cases and Gotchas

Bucketing finer than the scrape interval produces gaps

If the panel’s X bucket resolves to an interval smaller than your Prometheus scrape interval, some buckets contain no sample and render as empty cells — a striped, holey heatmap that looks like data loss. Always set a Min step on the query at or above the scrape interval (15s here), and let the X bucket use $__interval so Grafana never asks for finer resolution than exists. If cells are still sparse, widen the dashboard time range or coarsen the scrape.

$__interval shrinks on zoom and re-buckets everything

$__interval is computed from the visible time range divided by panel width, so zooming in makes it smaller and zooming out makes it larger. A heatmap that looked clean over 6 hours can turn sparse when you zoom to 15 minutes, because the buckets shrank below the scrape interval. This is expected behavior, not a bug — but pin a Min step so the query resolution has a floor regardless of zoom, otherwise a tight zoom during an incident will make the panel go blank exactly when you need it.

Null gaps are not zero lag

When a replica’s exporter goes down, its lag metric goes absent — not to zero. If you configure the panel to render nulls as zero, a dead replica’s lane turns green (healthy) when it should read as a gap (unknown). Set null handling to leave gaps, and rely on a separate up == 0 state-timeline lane to distinguish “exporter down” from “lag is genuinely low.” Treating absence as zero is the single most dangerous misconfiguration on a lag panel because it hides outages behind a healthy color.

FAQ

Should I use a Heatmap panel or a state-timeline for replica lag?

Use a state-timeline when you want one clearly labelled row per replica and discrete SLA bands (green/yellow/red), which is the easiest to read for a bounded fleet. Use the Heatmap panel when you care about the continuous distribution of lag values over time across many replicas and want density rather than per-node rows. For spotting one lagging node in a fleet under about thirty replicas, the state-timeline is usually clearer.

Why are there gaps or white cells in my lag heatmap?

White or empty cells mean no data point landed in that time bucket, usually because the panel’s bucket interval is finer than the Prometheus scrape interval or a replica exporter was briefly down. Set a Min step on the query at or above the scrape interval, and set the panel to treat null values as connected or zero only if a missing sample genuinely means zero lag, which for a down exporter it does not.

How do I align heatmap color thresholds with my lag SLA?

Set the color scheme to Thresholds mode and place the steps at your SLO warning and critical seconds, for example green below 0.5s, yellow at 0.5s, red at 5s. Use the same numbers that drive your Prometheus alert rules so a red cell on the heatmap corresponds exactly to a firing alert, keeping the visual and the paging in agreement.

← Back to Grafana Dashboards for Read Replica Fleets