OpenTelemetry Span Attributes for Read Replica Routing Decisions

Problem statement: a trace shows a database call took 90 ms and was served by the primary, and there is no way to tell whether that was correct behaviour or a bug.

The difference between a descriptive span and a diagnostic one is a handful of attributes recorded at the moment the routing decision was made. This page specifies exactly which, where to set them, and how to keep them from wrecking your metric cardinality. It implements the decision layer described in tracing and SLO reporting for replica reads.

Symptom Identification

Your database spans are descriptive rather than diagnostic if:

  • A trace shows the node that served a query but not why that node was chosen.
  • You can see that primary reads are rising and cannot tell whether the cause is lag, transaction pinning, or a classification change.
  • The dashboard’s replica share and the traces you open disagree, and reconciling them takes an hour during an incident.
  • Database spans have a duration and little else — no server address, no role, no lag.
  • A metric added alongside the tracing has its own code path, so the two are computed differently and drift.

The second point is the practical cost. Without a reason attribute, a rise in primary reads is a mystery that requires reproducing the condition to investigate; with one, it is a single dashboard panel split by cause.

Root Cause Analysis

The routing reason exists only inside the router, only at the moment of selection. Once the query has run, nothing in the connection, the result, or the database’s own statistics records why this node was chosen. A span attribute set at decision time is the only durable record.

Instrumenting after the fact loses information and can be wrong. Reading the server address from the connection object after execution tells you where the connection points now, which may differ if the connection was recycled or the router failed over mid-request. And it cannot recover the reason at all.

Spans and metrics have opposite cardinality economics. A span is one sampled record; attaching a query digest, a user id, or a full SQL string to it costs a few bytes on the sampled fraction. A metric label multiplies the number of stored time series by the number of distinct values, forever. The same attribute that is free on a span can be catastrophic on a metric, which is why the two need explicitly separate attribute sets rather than a single shared one.

Deriving the metric from the span removes a whole class of incident confusion. When the dashboard number and the trace view are computed by different code, they disagree eventually, and the disagreement surfaces during an incident when nobody has time to work out which is right. One measurement rendered two ways cannot do this.

Two primary-served reads, distinguishable only by the reason attribute Two span records side by side. Both record the same operation, the same duration and the same primary node. The first carries a selection reason of fallback lag, meaning the freshness gate correctly rejected every replica; this is the system working. The second carries a selection reason of pinned transaction, meaning an eagerly opened transaction forced an ordinary read onto the primary; this is a defect costing replica capacity on every request. Without the reason attribute the two records are byte-identical. the same span, with and without db.replica.select_reason db.operation.name = SELECT server.address = pg-primary db.replica.role = primary duration = 92 ms (no reason) db.operation.name = SELECT server.address = pg-primary db.replica.role = primary duration = 92 ms (no reason) identical — and one of them is a bug identical — and one of them is not db.replica.select_reason = fallback_lag every replica was outside the freshness budget correct behaviour under stress — investigate capacity db.replica.select_reason = pinned_txn an eager transaction wrapper forced the primary a defect — costs replica capacity on every request One attribute, set at the point of decision, is the difference between an answer and a guess.

Step-by-Step Resolution

Step 1 — Instrument at the point of decision

python
from opentelemetry import trace
tracer = trace.get_tracer("db.router")

def execute_read(sql, params, *, path: str, freshness_budget_s: float):
    with tracer.start_as_current_span("db.query") as span:
        # The decision happens HERE. Every attribute below is knowable only
        # at this moment; none of it can be recovered after the query runs.
        choice = router.select(freshness_budget_s)

        _set_standard(span, choice)
        _set_routing(span, choice, path, freshness_budget_s)

        rows = choice.node.execute(sql, params)

        span.set_attribute("db.read.served_age_ms", choice.observed_age_ms())
        span.set_attribute("db.read.within_budget",
                           choice.observed_age_ms() <= freshness_budget_s * 1000)
        return rows

Inline verification: a trace exported from a development run shows all attributes present on the db.query span, including on the fast path where no fallback occurred.

Step 2 — Use conventional names plus a small routing set

python
def _set_standard(span, choice) -> None:
    """Semantic-convention attributes. Keep these names exactly — back ends
    group, filter and build service maps from them automatically."""
    span.set_attribute("db.system.name", "postgresql")
    span.set_attribute("db.namespace", choice.node.dbname)
    span.set_attribute("server.address", choice.node.host)
    span.set_attribute("server.port", choice.node.port)
    span.set_attribute("db.operation.name", "SELECT")

def _set_routing(span, choice, path: str, budget_s: float) -> None:
    """Routing attributes: no conventional equivalent exists, so they live
    under a clear prefix and are documented beside the router."""
    span.set_attribute("db.replica.role", choice.node.role)          # primary|replica
    span.set_attribute("db.replica.instance", choice.node.name)      # r1, r2, …
    span.set_attribute("db.replica.select_reason", choice.reason)    # closed set
    span.set_attribute("db.replica.lag_ms", choice.lag_ms)
    span.set_attribute("db.replica.candidates_rejected", choice.rejected_count)
    span.set_attribute("db.read.path", path)
    span.set_attribute("db.read.freshness_budget_ms", budget_s * 1000)

Inline verification: the trace back end’s attribute browser lists the conventional attributes under its built-in database section, and the db.replica.* attributes as custom ones.

Step 3 — Define the reason as a closed enumeration

python
from enum import StrEnum

class SelectReason(StrEnum):
    """Closed set. Adding a member is a deliberate change that must be
    reflected in the dashboards; an open-ended string is not."""
    ELIGIBLE          = "eligible"            # a replica met the budget
    FALLBACK_LAG      = "fallback_lag"        # all replicas outside the budget
    FALLBACK_NO_NODE  = "fallback_no_healthy" # no replica was up
    PINNED_TXN        = "pinned_txn"          # inside an open transaction
    PINNED_SESSION    = "pinned_session"      # session state made it ineligible
    PINNED_WATERMARK  = "pinned_watermark"    # unmet read-your-writes watermark
    FORCED_PRIMARY    = "forced_primary"      # explicit application hint
    UNCLASSIFIED      = "unclassified"        # classifier could not prove a read

The vocabulary is the design. Each member corresponds to a distinct cause with a distinct remedy — pinned_txn is a framework setting, fallback_lag is capacity, unclassified is a routing rule. A free-text reason produces a dashboard nobody can group.

Inline verification: len(SelectReason) <= 12 and every branch in the router assigns exactly one of them; a linter rule or a test asserting exhaustiveness keeps it that way.

Step 4 — Emit metrics from the same data

python
from opentelemetry import metrics
meter = metrics.get_meter("db.router")

read_age = meter.create_histogram(
    "db.read.served_age", unit="ms",
    description="Age of the data a read actually returned")

def _record(choice, path: str, within_budget: bool) -> None:
    # BOUNDED labels only. Everything here is a fixed vocabulary or a boolean.
    read_age.record(choice.observed_age_ms(), {
        "db.read.path": path,                       # fixed set of read paths
        "db.replica.role": choice.node.role,        # 2 values
        "db.replica.select_reason": choice.reason,  # 8 values
        "db.read.within_budget": str(within_budget).lower(),   # 2 values
    })

Because this records on every span while only a fraction of traces are exported, the aggregate is exact and the traces are exemplars — the standard answer to the cost objection.

Inline verification: the metric’s series count equals roughly paths × 2 × 8 × 2; anything far larger means an unbounded label slipped in.

Step 5 — Audit cardinality before deploying

promql
# Run against a staging deployment carrying representative traffic.
# Any metric above a few thousand series deserves an explanation.
topk(10, count by (__name__) ({__name__=~"db_.*"}))

# And the specific one this change adds:
count(count by (db_read_path, db_replica_role, db_replica_select_reason,
                db_read_within_budget) (db_read_served_age_ms_count))

Inline verification: the computed series count matches the product of the label vocabularies. A mismatch means a label is taking values you did not enumerate.

Configuration Snippet

yaml
# otel-collector: drop any high-cardinality attribute before it becomes a
# metric label, while leaving it on the span where it is harmless and useful.
processors:
  # Spans keep everything.
  attributes/spans: {}

  # Metrics get a strict allowlist.
  attributes/metrics:
    actions:
      - key: db.statement            # useful on a span, catastrophic on a metric
        action: delete
      - key: enduser.id
        action: delete
      - key: db.replica.lag_ms       # a continuous value — never a label
        action: delete

  # Belt and braces: cap series growth so a mistake degrades rather than
  # taking the metrics pipeline down.
  metricstransform/limit:
    transforms:
      - include: db.read.served_age
        action: update
        operations:
          - action: aggregate_labels
            label_set: [db.read.path, db.replica.role,
                        db.replica.select_reason, db.read.within_budget]
            aggregation_type: sum

service:
  pipelines:
    traces:
      processors: [attributes/spans, tail_sampling]
    metrics:
      processors: [attributes/metrics, metricstransform/limit]

The aggregate_labels operation is the safety net worth having: it explicitly reduces the metric to the four intended labels regardless of what the application sent. An instrumentation change that adds a new attribute then cannot silently multiply the series count — the collector drops it, the dashboard is unaffected, and the attribute remains available on traces where it belongs.

One attribute set, two destinations with opposite cardinality rules Attributes emitted by the router split into two paths. The span path accepts everything including the SQL statement, the user identifier and the numeric lag, because spans are sampled and each is stored as an individual record. The metric path passes through an allowlist that keeps only path, role, reason and a within-budget boolean, because every distinct label combination becomes a permanent time series. router emits role · instance · reason lag · statement · user path · budget · age span — keep everything sampled; each stored individually high cardinality is free here cost ∝ sampled volume 1 % of traces × 9 attributes metric — allowlist only path · role · reason · within_budget every combination is a series, forever cost ∝ product of vocabularies 12 × 2 × 8 × 2 ≈ 384 series add db.statement → millions Enforce the split in the collector, not only in the application — then an instrumentation mistake degrades instead of breaking.

Verification and Rollback

Confirm every span carries the routing set:

bash
# Query the trace back end for spans missing the reason attribute.
# In a healthy deployment this returns nothing.
otel-cli query --service checkout-api \
  --filter 'name = "db.query" AND db.replica.select_reason IS NULL' \
  --last 15m --limit 20

Missing attributes usually mean a second code path issues queries without going through the instrumented router — a background job, a migration runner, or a library with its own connection.

Confirm the metric matches the traces:

promql
# Share of primary-served reads from the metric.
sum(rate(db_read_served_age_ms_count{db_replica_role="primary"}[10m]))
  / sum(rate(db_read_served_age_ms_count[10m]))

Compare against the same ratio computed from sampled spans over the same window, scaled by the sampling rate. Agreement within the sampling error confirms the two are derived from the same events; a persistent gap means one path is instrumented and the other is not.

Rollback. Instrumentation is additive and reverts cleanly, but the two halves have different risk profiles:

yaml
instrumentation:
  span_attributes: true      # negligible cost; keep on
  derived_metrics: false     # the cardinality risk lives here

If a cardinality problem appears, disable the derived metrics first and keep the span attributes — you lose the aggregate but retain the ability to investigate individual requests. Removing the span attributes should be a last resort, because doing so makes the class of problem they exist to reveal invisible again, and the cost of carrying them is close to zero.

Edge Cases and Gotchas

Semantic conventions have changed names between versions

db.system became db.system.name, and net.peer.name became server.address, in the stabilised database conventions. Dashboards and processors keyed on the old names silently stop matching after an SDK upgrade. Pin the convention version your instrumentation targets, and when upgrading, emit both names for one release so back-end queries can be migrated without a gap.

A proxy in the path hides the real node

If routing happens in ProxySQL or HAProxy rather than in-process, the application sees only the proxy’s address and cannot populate db.replica.instance or the reason. Two partial remedies: have the application ask the server which node executed the statement (inet_server_addr()), which recovers the instance but not the reason; and export the proxy’s own routing decisions as metrics, joined to traces by time rather than by trace id. Neither is as good as in-process routing for observability, which is a real if secondary argument in the tool-choice discussion covered in choosing a read replica routing proxy.

lag_ms must never become a metric label

It is a continuous value, so as a label it creates one series per distinct millisecond observed — effectively unbounded. It belongs on the span, where it is valuable, and in a histogram if you want its distribution. The collector rule above deletes it from the metric path specifically because it is the most tempting mistake in this attribute set.

FAQ

Which attribute matters most?

The selection reason. Two spans that both show a read served by the primary are indistinguishable without it, yet one may be the freshness gate working correctly under stress and the other a transaction wrapper wasting replica capacity on every request. The reason is knowable only at the moment of the decision and is unrecoverable afterwards, so omitting it makes a whole class of problem permanently invisible.

Should I use the official semantic conventions?

For everything they cover, yes — db.system.name, server.address, db.namespace and the rest — because existing back-end tooling groups and filters on those names automatically. The routing attributes have no conventional equivalent, so define them under a clear prefix of your own such as db.replica and document the vocabulary next to the router code.

Where exactly should the attributes be set?

Inside the router function that selects the node, before the query executes. Setting them afterwards from the connection object loses the reason entirely and can attribute the wrong node if the connection was recycled. If routing happens in a proxy rather than in-process, the application can still record the node it observed, but the reason has to come from the proxy’s own logs and will be much harder to join up.

How do I stop these attributes exploding metric cardinality?

Separate what goes on the span from what goes on the metric. Spans can carry anything, including a query digest or a user identifier, because they are sampled and stored individually. Metrics derived from spans must use only bounded label sets — path, role, reason and a boolean — with every value drawn from a fixed list. A single unbounded label such as a raw SQL string multiplies your series count by the number of distinct queries.

← Back to Tracing & SLO Reporting for Replica Reads