Grafana Dashboards for Read Replica Fleets

A fleet of read replicas fails one node at a time. One replica falls behind on WAL replay, its pool saturates, and reads routed there start returning stale data — while every fleet-wide average stays green because the other nine nodes are healthy. A dashboard that only shows aggregates will never surface this. The job of a replica fleet dashboard is the opposite of a summary: it must make a single deviant node visible at a glance across ten or fifty replicas, and it must connect that visual signal to the same thresholds your paging rules fire on. This guide sits inside the broader work of monitoring and observability for read replicas and covers how to structure such a dashboard so it scales with the fleet instead of fighting it.

The two anchors are template variables (so one dashboard adapts to any fleet size without editing panels) and repeating rows (so per-replica detail is generated, not hand-copied). Layered on top are the RED and USE method applied to the replica query path and resource, thresholds bound to your lag SLA, and provisioning as code so the dashboard is reviewable and reproducible rather than clicked together in the UI.

Concept Definition and Scope

A read replica fleet dashboard is a single Grafana dashboard, backed by Prometheus, that renders both a fleet-wide summary and per-replica detail from one JSON model. “Fleet-wide” means aggregate panels — total read QPS, worst-case lag across all nodes, count of replicas in each health state. “Per-replica detail” means the same set of panels cloned once per node so you can drill from “something is wrong” to “replica-b is wrong” without leaving the page.

This is distinct from the metric pipeline that feeds it. The exporters, scrape config, and PromQL that expose replication lag and pool depth are covered in Prometheus metrics for replica health and lag; this page assumes those metrics already exist in Prometheus and focuses on how to visualize them. It is also distinct from paging: the alert rules that wake someone up live in alerting on replication lag and pool saturation. The dashboard and the alerts share threshold values but serve different moments — the dashboard is for investigation, the alert is for notification.

Scope for this guide: the panel taxonomy for a fleet, template-variable and repeat mechanics, threshold and annotation wiring to a lag SLA, provisioning the dashboard as code, and linking panels back to Alertmanager. The one long-form panel type — the lag heatmap — has its own step-by-step build in the section index at the end.

Mechanism Deep-Dive

The data flow behind every panel

Every panel on the dashboard resolves to a PromQL query against a Prometheus datasource, which in turn scrapes exporters running beside each replica. Understanding that chain matters because most “broken panel” incidents are actually breaks upstream — an exporter down, a relabel that dropped the replica label, a scrape interval too coarse for the panel’s resolution. The diagram traces a metric from the replica to the panel and shows where the SLA threshold and Alertmanager attach.

Template variables: one dashboard, any fleet size

The variable is what lets a single JSON model serve a three-node fleet and a fifty-node fleet without edits. Define a query-backed variable that enumerates replicas from a metric that is always present — up filtered to the replica job, not a lag metric that goes absent when a node dies.

Name:        replica
Type:        Query
Data source: Prometheus
Query:       label_values(up{job="postgres_replicas"}, instance)
Refresh:     On Time Range Change
Multi-value: on
Include All: on
Sort:        Alphabetical (asc)

Backing the variable with up rather than pg_replication_lag_seconds is deliberate: when a replica’s exporter goes down you still want it to appear in the variable list (so its state-timeline panel can show the outage), and a lag metric would have vanished from the series set. The Include All option feeds the fleet-summary row; the multi-value list feeds the drill-down.

Repeating rows: generate per-replica detail

A repeating row is a row whose repeat field is set to a variable. At render time Grafana clones the row — and every panel inside it — once per selected value, substituting $replica into each panel’s PromQL. You author three panels once; a ten-node fleet renders thirty. This is the single most important mechanic for fleet dashboards, because it decouples panel maintenance from fleet size.

{
  "type": "row",
  "title": "Replica: $replica",
  "repeat": "replica",
  "repeatDirection": "v",
  "collapsed": false,
  "panels": []
}

Inside the row, panels reference the variable in their targets, for example a lag time series scoped to the current clone:

pg_replication_lag_seconds{instance="$replica"}

Use repeatDirection: "v" (vertical) for fleets larger than a handful of nodes so each replica gets a full-width band; horizontal repeats get cramped past four columns.

RED and USE applied to a replica

Two method frameworks map cleanly onto a replica and together cover both “is the read service healthy” and “is the node healthy.” Apply RED to the query path the replica serves — Rate (read QPS hitting this replica), Errors (query cancellations, connection failures), Duration (query latency distribution). Apply USE to the replica as a resource — Utilization (CPU, disk IOPS, pool checkout ratio), Saturation (apply queue depth, pool wait count), Errors (WAL apply errors, exporter scrape failures).

The distinction matters when you triage. A RED spike in Duration with flat USE says the query mix changed. A USE spike in Saturation with flat RED says replay or I/O is falling behind before it has hit user-visible latency — the earliest actionable signal, and the one a naive latency-only dashboard misses entirely.

Trade-Off Comparison Table

The practical layout is a hybrid: a fleet-summary row at the top (worst-lag single-stat, pool gauges, a lag heatmap across all nodes), then a repeating per-replica row for drill-down. The heatmap answers “which node,” the repeating row answers “why.”

Configuration Runbook

Provisioning the datasource

Provision the Prometheus datasource as code so the dashboard’s datasource references resolve identically in every environment. Grafana loads this at startup from its provisioning directory.

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.monitoring.svc:9090
    uid: prom-main        # reference this uid from panel targets, not the name
    isDefault: true
    jsonData:
      timeInterval: "5s"  # match the replica scrape interval
      httpMethod: POST

Pinning a stable uid matters: exported dashboard JSON references the datasource by uid, so a hardcoded uid keeps a version-controlled dashboard portable across staging and production.

Provisioning the dashboard

Never build the production dashboard by clicking in the UI and leaving it in Grafana’s database. Export the JSON model and load it through a dashboard provider so it is immutable, reviewable, and reproducible.

# /etc/grafana/provisioning/dashboards/replica-fleet.yaml
apiVersion: 1
providers:
  - name: replica-fleet
    orgId: 1
    folder: "Database Replicas"
    type: file
    disableDeletion: true
    editable: false        # UI edits are discarded on reload — JSON is source of truth
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards/replica-fleet

Drop the exported replica-fleet.json into that path. Setting editable: false forces every change through the JSON in version control, which is what keeps dashboard thresholds and alert thresholds from silently drifting apart.

Panel target definitions

The core per-replica panels reduce to a small set of PromQL targets. These assume the metric names from the replica health and lag exporter setup; adjust names to your exporter.

# Lag time series (seconds) — per replica
pg_replication_lag_seconds{instance="$replica"}

# WAL apply rate (bytes/sec) — how fast the replica drains its queue
rate(pg_replication_replay_lsn_bytes{instance="$replica"}[1m])

# Pool saturation (fraction of max) — PgBouncer / app pool
pgbouncer_pools_server_active_connections{instance="$replica"}
  / pgbouncer_pools_server_maxwait_connections{instance="$replica"}

# Replica up/down for the state-timeline panel
up{job="postgres_replicas", instance="$replica"}

# Fleet-wide worst lag (summary single-stat, no $replica filter)
max(pg_replication_lag_seconds{job="postgres_replicas"})

The pool-saturation query is the fleet’s early-warning signal for read overload; its provenance and the exporter that produces it are covered in monitoring PgBouncer pool saturation with Prometheus.

Thresholds tied to the lag SLA

Field thresholds turn a raw number into a color the on-call eye reads instantly. Bind them to the same SLO tiers you route traffic on. If your Tier-1 SLA is 500 ms lag, the warning band starts there and critical at the point where you exclude the replica from routing.

{
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green",  "value": null},
          {"color": "yellow", "value": 0.5},
          {"color": "red",    "value": 5}
        ]
      }
    }
  }
}

Add a threshold annotation so the moment a replica crossed the SLA is drawn as a vertical marker on the time axis, correlating the breach with deploys or bulk loads:

{
  "annotations": {
    "list": [
      {
        "name": "SLA breach",
        "datasource": {"uid": "prom-main"},
        "enable": true,
        "expr": "pg_replication_lag_seconds{job=\"postgres_replicas\"} > 5",
        "iconColor": "red",
        "titleFormat": "Replica over lag SLA"
      }
    ]
  }
}

These same 0.5 and 5 values must appear in the Prometheus alert rules; the way to guarantee that is to template both from one variables file, described below.

Monitoring and Alerting Signals

The dashboard is where investigation happens, but it should hand off cleanly to paging. Three wiring points make that handoff work.

Panel data links to the alert view. Add a data link on each lag panel that jumps to Grafana’s alerting page filtered to that replica, so an on-call engineer moves from graph to firing rule in one click.

{
  "links": [
    {
      "title": "Firing alerts for $replica",
      "url": "/alerting/list?dataSource=prom-main&queryString=instance%3D$replica",
      "targetBlank": true
    }
  ]
}

Shared thresholds. The panel threshold and the alert rule expression must use identical numbers. Generate both from one source:

# slo.vars.yaml — single source of truth for every threshold
replica_lag_warning_seconds: 0.5
replica_lag_critical_seconds: 5
pool_saturation_warning: 0.75
pool_saturation_critical: 0.90

A small templating step renders this into the dashboard JSON thresholds and into the Prometheus rule expr, so bumping the SLA updates the panel color and the paging rule in the same commit. The rule catalog that consumes these values lives in alerting on replication lag and pool saturation, and the specific SLO-burn rules in Alertmanager rules for replica lag SLO burn.

Signals worth a panel and an alert both:

• Per-replica lag crossing the warning and critical SLO bands — the primary routing signal, mirrored from the same telemetry that drives detecting and handling replication lag in real-time.
• Pool saturation above 75% (warning) and 90% (critical) — read overload precedes lag.
• Apply rate dropping toward zero while WAL receive continues — replay stall, visible before wall-clock lag climbs.
• up == 0 for the replica job — exporter or node down; must be alertable on absence, not just graphable.

Failure Modes and Recovery Steps

1. Repeating row renders empty for a live replica

Symptom: A replica you know is up has no row on the dashboard. Root cause: the template variable is backed by a metric that goes absent (a lag metric) or the variable’s refresh is set to “On Dashboard Load” and the fleet changed since load.

Recovery:

1. Repoint the variable query to label_values(up{job="postgres_replicas"}, instance).
2. Set variable refresh to On Time Range Change.
3. Confirm the instance label survives relabeling in the scrape config; a dropped label breaks the $replica substitution silently.

2. Fleet averages hide one lagging node

Symptom: Every summary panel is green while users report stale reads. Root cause: aggregate panels use avg() which dilutes a single outlier across the fleet.

Recovery:

1. Change fleet-summary lag to max(pg_replication_lag_seconds) not avg().
2. Add the lag heatmap so per-node distribution is always visible, not just the aggregate.
3. Keep the repeating per-replica row below the summary so a red node is one scroll away.

3. Dashboard threshold and alert threshold disagree

Symptom: A panel is red but nothing paged, or vice versa. Root cause: thresholds were edited in the Grafana UI while the alert rule kept the old value.

Recovery:

1. Set the dashboard provider editable: false so UI edits are discarded on reload.
2. Move both thresholds into slo.vars.yaml and template them into the dashboard JSON and the alert rules.
3. Redeploy both from the same commit; verify the panel step values equal the rule expr constants.

4. Panels stall or error under many repeats

Symptom: A fifty-node fleet dashboard times out or shows “too many outstanding requests.” Root cause: fifty repeated rows issue hundreds of concurrent PromQL queries per refresh.

Recovery:

1. Raise the dashboard refresh interval to 30s or 1m — replica lag does not need 5s dashboard refresh even if scraped at 5s.
2. Collapse the repeating row by default (collapsed: true) so its panels only query when expanded.
3. For large fleets, prefer the single lag heatmap as the always-on view and reserve repeats for on-demand drill-down.

5. State-timeline shows constant flapping

Symptom: The up/down panel is a barcode of state changes. Root cause: scrape timeouts or exporter GC pauses produce transient up == 0 samples that are not real outages.

Recovery:

1. Compare against avg_over_time(up[1m]) for the timeline instead of the raw instant to smooth single-sample gaps.
2. Investigate exporter latency; a slow exporter response past scrape_timeout reads as down.
3. Reserve real alerting for up == 0 for: 1m so momentary blips do not page.

Section Index

The following guide goes deep on the one panel that most repays careful construction on a fleet dashboard.

Fleet Lag Visualization

Building a Replica Lag Heatmap in Grafana Step-by-step construction of the heatmap panel that shows lag distribution across the entire fleet over time — the fastest way to spot a single lagging replica among many. Covers the PromQL that produces per-replica lag buckets, configuring the Heatmap versus state-timeline panel, mapping color thresholds to your lag SLA, reading the heatmap during an incident, and the bucketing, $__interval, and null-gap gotchas that trip up first builds.

FAQ

Related

← Back to Monitoring & Observability for Read Replicas

• Building a Replica Lag Heatmap in Grafana — the step-by-step build of the single most useful fleet-wide panel, including the PromQL buckets and color-to-SLA mapping.
• Prometheus Metrics for Replica Health and Lag — the exporters, scrape config, and metric names that every panel on this dashboard queries.
• Alerting on Replication Lag and Pool Saturation — the paging rules that share thresholds with these dashboard panels and route through Alertmanager.
• Detecting and Handling Replication Lag in Real-Time — the measurement techniques and heartbeat design behind the lag metrics these panels visualize.
• Monitoring & Observability for Read Replicas — the parent section covering the full metrics, dashboards, alerting, and query-analysis stack for a replica fleet.