Monitoring PgBouncer Pool Saturation with Prometheus

Problem statement: Application connections to your read replicas start timing out with query_wait_timeout, but the replicas themselves look healthy — the bottleneck is the PgBouncer pool in front of them, and without Prometheus visibility into cl_waiting, sv_active, and maxwait you cannot see the saturation building until clients are already failing.

Symptom Identification

Pool saturation announces itself through the application and the pooler long before the database shows any stress:

  • Application logs fill with server closed the connection unexpectedly or driver-level connection-acquisition timeouts, while replica CPU and lag remain nominal.
  • SHOW POOLS in the PgBouncer admin console shows cl_waiting climbing above zero and staying there, with sv_active pinned at default_pool_size.
  • maxwait — the seconds the oldest waiting client has been blocked — creeps toward your configured query_wait_timeout.
  • Latency p99 rises on read endpoints even though per-query execution time on the replica is flat, because the time is being spent waiting for a backend, not running the query.

The tell that distinguishes this from a database problem is that replica-side metrics are healthy. The queue is forming in the pooler, and only pooler metrics reveal it.

Root Cause Analysis

PgBouncer multiplexes many client connections onto a bounded set of backend server connections per pool. When every server connection in a pool is checked out, additional client requests are parked in a wait queue until a backend is released. Three counters describe this exactly:

  • cl_waiting — clients currently blocked waiting for a server connection. Zero in a healthy pool; any sustained non-zero value means demand exceeds default_pool_size.
  • sv_active — server connections currently assigned to a client and doing work. This rises to default_pool_size under load and stays there; on its own it is a utilisation signal, not a saturation signal.
  • maxwait — how long, in seconds, the oldest waiting client has been queued. This is the direct precursor to query_wait_timeout errors: when maxwait reaches the timeout, clients start being dropped.

Saturation is the combination of sv_active at the pool ceiling and cl_waiting above zero. High sv_active alone is a fully-utilised pool serving traffic efficiently — the desired state at peak. The mistake teams make is alerting on utilisation and paging on every busy period, or conversely watching only the database and missing that the pool is the constraint.

The exporter’s job is to lift these counters out of the transient SHOW POOLS snapshot and into a time series you can rate, quantile, and alert on. This sits inside the broader Prometheus metrics for replica health and lag instrumentation, and the pool it observes is configured per configuring PgBouncer for read-only connection pools.

Architecture: Where the Queue Forms

PgBouncer pool saturation and where the wait queue forms Multiple application clients connect to a PgBouncer pool. Active clients hold server connections; when all server connections are busy, further clients enter a wait queue measured by cl_waiting and maxwait. The pool's server connections route to the read replica. A pgbouncer_exporter reads SHOW POOLS, STATS, and LISTS from the admin database and exposes the counters to Prometheus. client (active) client (active) client (waiting) client (waiting) cl_waiting • maxwait PgBouncer pool default_pool_size = 25 sv_active (busy backends) sv_idle (free backends) Read replica max_connections budget pgbouncer_exporter SHOW POOLS / STATS / LISTS Prometheus

Step-by-Step Resolution

Step 1: Expose the admin database to a stats user

PgBouncer serves its counters from a virtual database named pgbouncer. Grant a dedicated stats account access without full admin rights.

ini
# pgbouncer.ini
[pgbouncer]
stats_users = stats_collector      ; can run SHOW commands, cannot RELOAD/SHUTDOWN
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/userlist.txt

Add the stats user to userlist.txt and reload:

bash
kill -HUP $(pgrep pgbouncer)

Inline verification: psql -p 6432 -U stats_collector pgbouncer -c "SHOW POOLS;" returns a row per pool with cl_active, cl_waiting, sv_active, and maxwait columns.

Step 2: Run pgbouncer_exporter

bash
pgbouncer_exporter \
  --pgBouncer.connectionString="postgres://stats_collector@localhost:6432/pgbouncer?sslmode=disable" \
  --web.listen-address=:9127

Inline verification:

bash
curl -s localhost:9127/metrics | grep -E 'pgbouncer_pools_(client_waiting|server_active)|maxwait'

You should see per-database, per-user series such as pgbouncer_pools_client_waiting_connections{database="appdb_ro"}.

Step 3: Scrape and label the pooler

yaml
# prometheus.yml
scrape_configs:
  - job_name: pgbouncer
    scrape_interval: 5s
    static_configs:
      - targets: ['pgb-1:9127']
        labels: {role: pooler, cluster: orders, az: use1-az1}
      - targets: ['pgb-2:9127']
        labels: {role: pooler, cluster: orders, az: use1-az2}

Inline verification: up{job="pgbouncer"} returns 1 for each pooler in the Prometheus expression browser.

Step 4: Compute the saturation PromQL

text
# Fraction of the pool that is actively serving, per pool
pgbouncer_pools_server_active_connections
  / on(instance, database, user) group_left
    (pgbouncer_config_default_pool_size)

# Backpressure: clients currently queued
pgbouncer_pools_client_waiting_connections

# Oldest client wait in seconds (the direct timeout precursor)
pgbouncer_pools_client_maxwait_seconds

Inline verification: under synthetic load (e.g. pgbench -c 100), the saturation ratio should approach 1.0 and client_waiting should rise while the ratio is pinned.

Step 5: Alert before clients time out

yaml
# rules/pgbouncer_saturation.yml
groups:
  - name: pgbouncer_saturation
    rules:
      - alert: PgBouncerPoolSaturated
        expr: |
          pgbouncer_pools_client_waiting_connections > 0
          and pgbouncer_pools_server_active_connections
              >= pgbouncer_config_default_pool_size
        for: 1m
        labels: {severity: warning}
        annotations:
          summary: "Pool {{ $labels.database }} on {{ $labels.instance }} saturated: clients queueing"

      - alert: PgBouncerClientWaitHigh
        # query_wait_timeout defaults to 120s; page at half of it
        expr: pgbouncer_pools_client_maxwait_seconds > 60
        for: 30s
        labels: {severity: critical}
        annotations:
          summary: "Oldest client on {{ $labels.database }} waited {{ $value }}s for a backend"

Inline verification: silence-test by lowering the threshold temporarily and confirming the alert transitions to firing in the Prometheus Alerts tab.

Configuration Snippet

A complete recording-rule block that turns the raw counters into a single saturation series dashboards can read cheaply:

yaml
# rules/pgbouncer_recording.yml
groups:
  - name: pgbouncer_recording
    interval: 15s
    rules:
      - record: pgbouncer:pool_saturation_ratio
        expr: |
          pgbouncer_pools_server_active_connections
            / clamp_min(pgbouncer_config_default_pool_size, 1)

      - record: pgbouncer:client_wait_p99_seconds
        expr: |
          max by (instance, database) (pgbouncer_pools_client_maxwait_seconds)

      - record: pgbouncer:avg_query_seconds:rate5m
        expr: |
          rate(pgbouncer_stats_total_query_time_seconds[5m])
            / clamp_min(rate(pgbouncer_stats_total_query_count[5m]), 1)

Verification and Rollback

Confirm the pipeline end to end and prove the alert fires under real saturation:

bash
# Drive load through the pool and watch the queue form
pgbench -h localhost -p 6432 -U app_read_user -d appdb_ro -c 120 -T 60 -S &

# In parallel, watch the raw counters
watch -n1 'psql -p 6432 -U stats_collector pgbouncer -c "SHOW POOLS;"'

While the load runs, pgbouncer:pool_saturation_ratio should approach 1.0 and PgBouncerPoolSaturated should move to pending, then firing.

If the exporter or a bad rule destabilises Prometheus, roll back the rule file and reload:

bash
git checkout HEAD -- rules/pgbouncer_saturation.yml rules/pgbouncer_recording.yml
curl -X POST http://localhost:9090/-/reload

If the saturation itself is the emergency, the immediate mitigation is to raise default_pool_size — but only within the replica’s max_connections budget, or you trade a pooler queue for remaining connection slots are reserved errors on the replica.

Edge Cases and Gotchas

maxwait resets, so rate() lies on it

maxwait is the wait time of the current oldest waiting client; the moment that client is served, the value drops to whatever the next-oldest client’s wait is, or to zero. It is a gauge with sawtooth behaviour, not a monotonic counter, so never wrap it in rate() or increase(). Alert on its instantaneous value with a for: duration to require the condition to persist, and use max_over_time(pgbouncer_pools_client_maxwait_seconds[5m]) if you want the worst wait in a window.

Per-pool cardinality with many databases

pgbouncer_exporter emits one series per (database, user) pool. A pooler fronting hundreds of database aliases or users produces a series set that multiplies across every counter. Keep the alias list lean, and if you must front many databases, drop the low-value per-pool counters at the scrape and keep only client_waiting, server_active, and maxwait. This is the same cardinality discipline described in the replica health metrics overview.

Session pooling hides transient saturation

In pool_mode = session, a backend is pinned to a client for the entire session, so a saturated pool clears only as sessions disconnect — which can be minutes. A brief cl_waiting spike you would shrug off in transaction mode signals a much longer stall in session mode. If your read pool runs session mode, lower the PgBouncerClientWaitHigh threshold and treat any sustained cl_waiting as urgent, because the queue drains slowly.

FAQ

What is the single best PgBouncer saturation signal?

Client waiting connections, cl_waiting, is the clearest signal. It counts clients that have asked for a server connection and are blocked because every backend in the pool is busy. A steady non-zero cl_waiting means default_pool_size is too small for the offered load, and it precedes the query_wait_timeout errors your application will see. Pair it with maxwait, which reports how long the oldest waiting client has been blocked in seconds.

Why is sv_active near pool size but cl_waiting still zero?

That is a fully utilised but not yet saturated pool: every server connection is doing useful work and clients are being served without queueing. It is the healthy high-load state. Saturation only begins when sv_active is pinned at the pool size AND cl_waiting rises above zero, meaning new clients cannot get a backend. Alert on the combination, not on high sv_active alone, or you will page on normal peak traffic.

Does transaction pooling change which metrics matter?

Yes. In transaction mode a backend is returned to the pool after every transaction, so sv_active turns over rapidly and a small pool serves many clients. Waiting is therefore driven by transaction duration, not session lifetime, and maxwait spikes point to slow queries on the replica rather than too many idle sessions. In session mode a backend is pinned for the whole client session, so cl_waiting reflects concurrent session count and a saturated pool clears much more slowly.

← Back to Prometheus Metrics for Replica Health and Lag