Designing Read Paths That Tolerate Seconds of Staleness

Problem statement: the replicas are a few hundred milliseconds behind, which is entirely normal, and users are reporting that their changes “did not save”.

Nothing is broken. What is happening is that a handful of screens read immediately after a write and expect the replica to know about it. This is a product-design problem more than an infrastructure one, and the design fixes are cheaper and more reliable than trying to eliminate lag. This page covers those fixes. The consistency models behind them are in eventual consistency patterns for read-heavy workloads.

Symptom Identification

  • Users report changes “not saving” while the database shows the write committed successfully.
  • Complaints cluster on flows that write and then immediately navigate to a page showing the written value.
  • A support playbook exists that says “ask them to refresh”.
  • Read-after-write bugs are reported far more often after deploys that increased write volume, because lag rose with it.
  • Nobody complains about the follower counts, activity feeds, or dashboards that are equally stale.

That last contrast is the whole insight. Staleness is imperceptible on most screens and highly visible on a few, and the few share one shape: the user just acted, and the next screen is supposed to reflect the action.

Root Cause Analysis

Users do not perceive stale data; they perceive lost actions. A three-second-old follower count is invisible. A form submission followed by a page that does not show the submission reads as failure — the user’s mental model is “I did something and it did not work”, not “the data is slightly behind”. This is why the design effort belongs almost entirely on post-write reads.

The re-read after a write is usually unnecessary. The common shape is: submit the form, receive an acknowledgement, then issue a fresh read to render the result page. But the acknowledged write already contains everything needed to render — the server accepted those values and confirmed them. Re-reading trades a guaranteed-correct local value for a possibly-stale remote one, and gains nothing.

Read-then-write is the case where staleness causes real harm. If a stale read decides a write — “the balance is 100, so subtract 30” — the write can be wrong, and a wrong write is far harder to recover from than a wrong display. Conditional writes remove the dependency entirely: UPDATE ... WHERE balance >= 30 is correct whatever a preceding read showed.

Showing age reframes staleness as information. A dashboard number with no timestamp is asserted as current, so being wrong is a defect. The same number labelled “as of 12:04” is a measurement, and users treat it accordingly. This costs one line of interface and removes an entire category of complaint.

Re-reading after a write versus echoing the acknowledged write Two sequences. In the first, the user submits a form, the primary acknowledges the write, the application issues a read against a replica that has not yet replayed it, and the result page renders the old value; the user concludes the change was lost. In the second, the application renders directly from the acknowledged write payload, so the result page is correct with no read at all, and the replica catches up unobserved in the background. re-read after write — the reported bug submit form primary commits 200 OK read from replica has not replayed yet renders the OLD value user: “it did not save” echo the acknowledged write — no read at all submit form primary commits returns the row no read issued renders the NEW value correct, and faster replica catches up, unobserved The second flow is one fewer round trip AND correct. Read-after-write is often a habit rather than a requirement.

Step-by-Step Resolution

Step 1 — Classify each read by what a stale answer costs

yaml
# read-tolerance.yml — reviewed with product, not only with engineering.
reads:
  - path: GET /feed
    tolerance: high            # seconds of staleness is imperceptible
    treatment: replica

  - path: GET /orders/{id}        # reached immediately after placing an order
    tolerance: none
    treatment: echo_the_write     # render from the acknowledged payload

  - path: GET /dashboard/metrics
    tolerance: high
    treatment: replica + show_as_of   # display the age

  - path: POST /wallet/withdraw     # a read decides a write
    tolerance: none
    treatment: conditional_write      # never read-then-write

Inline verification: every read in the application appears exactly once, and the none-tolerance list is short — typically under a tenth of all reads.

Step 2 — Echo the local write

python
# Before: the result page re-reads and may see the pre-write state.
@app.post("/profile")
def update_profile(form):
    db.execute("UPDATE profiles SET name = %s WHERE id = %s", form.name, uid)
    profile = db.query("SELECT * FROM profiles WHERE id = %s", uid)   # ← replica
    return render("profile.html", profile=profile)

# After: the write returns what it wrote, and the page renders from that.
@app.post("/profile")
def update_profile(form):
    profile = db.query_one(
        "UPDATE profiles SET name = %s WHERE id = %s RETURNING *",     # ← primary
        form.name, uid)
    return render("profile.html", profile=profile)

RETURNING * makes this free: the write already touched the row, so returning it costs nothing and removes a round trip as well as the correctness problem.

Inline verification: with replay paused on every replica, the profile page still renders the new name immediately after submission.

Step 3 — Show data age where the value is time-sensitive

python
# The read path already computes the age; surface it rather than hiding it.
metrics, age_s = replica.query_with_lag(DASHBOARD_SQL)
return render("dashboard.html", metrics=metrics, as_of=now() - age_s)
html
<!-- One line of interface that converts a wrong number into an old number. -->
<p class="metric">{{ metrics.active_users }}</p>
<p class="as-of">as of {{ as_of | time }} · refreshes every 30 s</p>

Inline verification: during an injected lag event the timestamp visibly falls behind, which is the intended behaviour — the interface is telling the truth about its own freshness.

Step 4 — Replace read-then-write with a conditional write

python
# Wrong: a stale read decides the write. If the replica is behind, the
# balance check passes against data that no longer reflects reality.
balance = replica.query_one("SELECT balance FROM wallets WHERE id = %s", wid)
if balance >= amount:
    primary.execute("UPDATE wallets SET balance = balance - %s WHERE id = %s",
                    amount, wid)

# Right: the precondition is evaluated on the primary, atomically with the
# write. A stale read cannot make this incorrect because there is no read.
rows = primary.execute(
    """UPDATE wallets SET balance = balance - %s
        WHERE id = %s AND balance >= %s
    RETURNING balance""", amount, wid, amount)
if not rows:
    raise InsufficientFunds()        # the precondition failed, atomically

Inline verification: run the withdrawal concurrently from two sessions against the same wallet; exactly one succeeds, regardless of replica lag.

Step 5 — Drill it with injected lag

bash
# The most valuable half hour available: pause replay and use the product.
for h in replica-1 replica-2; do
  psql -h $h -c "SELECT pg_wal_replay_pause()"
done

echo "Now click through: sign up, update a profile, place an order,"
echo "check a balance, view the dashboard. Note anything that looks BROKEN"
echo "rather than merely old."

for h in replica-1 replica-2; do
  psql -h $h -c "SELECT pg_wal_replay_resume()"
done

Inline verification: the list of screens that looked broken matches the tolerance: none entries in step 1. Anything on the list that is not in the file is a gap in the classification.

Configuration Snippet

python
# read_treatments.py — the four treatments as explicit, named helpers, so a
# code reviewer can see which guarantee a given read path is claiming.

def replica_read(sql, *args):
    """Default. Staleness is imperceptible for this value."""
    return replica.query(sql, *args)

def echo_write(sql_returning, *args):
    """Post-write render. The write returns what it wrote; no read follows."""
    return primary.query_one(sql_returning, *args)

def aged_read(sql, *args):
    """Time-sensitive value shown with its age, so staleness is information."""
    rows, lag_s = replica.query_with_lag(sql, *args)
    return rows, time.time() - lag_s          # (value, as_of)

def conditional_write(sql_with_precondition, *args):
    """A write whose precondition is evaluated atomically on the primary.
    Never preceded by a read that decides whether to call it."""
    rows = primary.execute(sql_with_precondition, *args)
    if not rows:
        raise PreconditionFailed()
    return rows
python
# A lint rule worth having: flag any handler that performs a write and then
# a read of the same table within the same request. That shape is almost
# always an echo_write opportunity and occasionally a correctness bug.
FORBIDDEN_SHAPE = """
    write to table T, then read from table T, same request, no echo_write
"""

Naming the four treatments is the load-bearing part. When every read is just db.query(...), the guarantee a path relies on is invisible and drifts silently as code is edited. When a path calls echo_write or aged_read, a reviewer can see the claim being made and challenge it, and a change from one to another becomes a visible diff rather than an accident.

The linting shape is worth building even crudely. Write-then-read-same-table within one request is the signature of the bug in step 2, and it is mechanically detectable from a query log — detecting accidental primary reads in production traffic collects the data you need to find it.

How a typical application's reads distribute across the four treatments A stacked bar of all read paths in an application. The large majority, around eighty percent, tolerate staleness entirely and go to replicas untouched. About ten percent are time-sensitive and use a replica with a visible as-of timestamp. Around seven percent are post-write renders that echo the acknowledged write. A remaining three percent are preconditions for writes and become conditional writes on the primary. A note observes that only the last ten percent required design work. all read paths in a typical application, by treatment replica, untouched — ≈ 80 % staleness is imperceptible; no design work needed aged echo cond. aged read — ≈ 10 % show an as-of timestamp echo the write — ≈ 7 % render from RETURNING conditional write — ≈ 3 % precondition on the primary The design effort is entirely in the right-hand fifth of the bar. Teams that report eventual consistency as intractable usually have not classified, so they are treating all reads as if they were in that fifth. Classification is a half-day exercise and it is what makes the rest tractable.

Verification and Rollback

Confirm the classification is complete by drilling rather than reasoning:

bash
# Pause replay, then walk every flow in the product with a checklist.
# Anything that looks BROKEN (not merely old) must appear in the
# tolerance: none list, or the classification has a gap.
psql -h replica-1 -c "SELECT pg_wal_replay_pause()"
# ... manual or automated walkthrough ...
psql -h replica-1 -c "SELECT pg_wal_replay_resume()"

Confirm read-after-write complaints actually fell:

promql
# Support tickets or in-product error reports tagged for this class.
sum(rate(support_tickets_total{category="change_not_saved"}[7d]))

This is the only metric that genuinely measures whether the work succeeded, and it lags the deploy by days. Watch it for two weeks rather than declaring victory on the day.

Rollback. Each treatment reverts independently and none carries data risk:

  • Echo the write — revert to the re-read if RETURNING proves awkward for a particular query. The re-read is correct whenever the replica is caught up, so the failure mode returns rather than worsens.
  • Aged reads — the timestamp is presentational; remove it without touching the data path.
  • Conditional writes — the one to keep. Reverting to read-then-write reintroduces a correctness bug rather than a display one, so treat a request to revert it as a signal that the conditional form was written incorrectly rather than that the pattern is wrong.

Edge Cases and Gotchas

Redirect-after-post crosses a request boundary

The standard post/redirect/get pattern means the render happens in a different request from the write, so the acknowledged payload is gone by the time you need it. Carry the written values in the flash session, or attach a write watermark to the redirect and have the follow-up request wait briefly for a replica to reach it — the mechanism in using application-level timestamps to bypass stale replicas.

Aggregates cannot be echoed

Echoing works for the row you wrote. It does not work for a count, a sum, or a ranking that the write affects, because you would have to recompute the aggregate to know its new value. For those, either accept the staleness and show the age, or adjust the displayed aggregate locally by the known delta — the second is fine for a counter and wrong for anything with a non-trivial computation.

Multi-device users see the un-echoed path

Echoing fixes the device that performed the write. The same user’s other tab or phone still reads from a replica and sees the old value, so a user watching two screens will observe them disagree for a second. This is usually acceptable; where it is not, the read needs a genuine watermark check rather than an echo, because there is no local write to echo on the second device.

FAQ

How do users actually perceive replication lag?

Almost never as old data, and almost always as a lost action. Nobody notices a follower count three seconds behind. Everybody notices submitting a form and landing on a page that does not show what they submitted, because that reads as the system having failed rather than being slightly behind. Design effort belongs on the second case, and it is a much smaller set of screens than it first appears.

Is echoing the local write the same as lying to the user?

No, provided the write has been acknowledged by the primary. At that point the value is durable and committed; rendering it is showing the truth slightly before the replica knows it. It becomes a lie only if you render optimistically before acknowledgement — a different pattern with different failure handling, and one that needs a visible correction path when the write fails.

When should data age be shown to users?

When the value is time-sensitive and the user might act on it — a dashboard metric, a stock level, a balance. A visible as-of timestamp changes the interpretation from a wrong number to an old number, which people accept readily. Do not show it on values where staleness is imperceptible; a timestamp on a stable value invites doubt about data that was never in question.

How do I stop a stale read causing a wrong write?

Do not let a read decide a write. Replace read-then-write flows with conditional writes on the primary — an UPDATE whose WHERE clause encodes the precondition, or an atomic increment rather than read-modify-write. Then a stale read can produce a wrong display, which is recoverable, but never a wrong write, which frequently is not.

← Back to Eventual Consistency Patterns for Read-Heavy Workloads