33 · Production Incidents, Root-Caused

The pattern across all seven

Six of the seven share a shape: the failing component stayed quiet. No crash, no error, no alert. Silence was the symptom.

1. The Kafka retry that could never happen

Symptom. A new consumer in GlobalChat logged attempt 2 of 5 once, and then nothing. Ever.

First theory. The handler returns an error, the offset stays uncommitted, Kafka redelivers. I had written exactly that into a comment and a commit message.

Evidence. Read the loop. ConsumeClaim logged the handler error and called continue. That did not mark the failed message, which looks like "leave it for redelivery". But the loop also moved on, and the next message that succeeded called MarkMessage on a higher offset. Committing offset N+1 commits everything below it. The failed message was skipped permanently. The "attempt 2" in the log had come from a process restart, not a redelivery. The match bridge's bounded retry and dead-letter path could only ever reach attempt 1.

Fix. consumeOne now retries in place: six attempts, backoff from 500 ms doubling to a 4 s cap, chosen so a handler's own five-attempt budget completes inside one session. Then it marks and moves on. A session that ends mid-retry returns without marking, because a rebalance genuinely does redeliver.

Rule. "Returning an error leaves the offset uncommitted" is a claim about your consumer loop, not a Kafka guarantee. Offsets are a high-water mark.

2. The refresh token that was "reused"

Symptom. Users bounced to a generic 401, "Invalid refresh token", after an outage. A spec was written blaming the restart for invalidating tokens, and proposing a Postgres token store and key stabilisation to fix it.

Evidence. The logs disproved every premise. Refresh tokens live in Redis with AOF on. JWT signing keys load from files and are never generated at boot. A 60 second RefreshReuseLeeway already existed, with its own test. The real chain: memory pressure, a thrashing box, a hard reboot. Minutes before it, a RefreshTokens call succeeded server-side while the tunnel died. The browser never received the new token. Twenty minutes later it replayed the old one, far past the leeway, and reuse detection did exactly what it is designed to do: revoke the whole session chain.

Fix. The frontend stopped hanging on it. The real bug there was a gate, !bootstrapped || !currentUser, that rendered nothing forever after bootstrap finished signed out. Now an expired session redirects to login with a reason in under a second, with a 10 s bootstrap watchdog and a 4 s auth RPC timeout. The backend change is named for what it is: a reuse policy decision (treat a replay of R1 as a lost response while R2 is still unused), not a storage project.

Rule. Any outage longer than the leeway that swallows a rotation response kills that session, and no client retry can fix it. Rotation plus reuse detection has a failure mode you must decide on, not discover.

3. The gateway that was "active" and silent

Symptom. Every web login failed for sixteen minutes. The site itself served 200 throughout.

First theory. The web client. Then a deploy.

Evidence. AuthGateway's identity store is Cassandra via gocql. When Cassandra stopped answering, the process stayed active. Nothing exited, systemd restarted nothing, and the structured log went completely quiet. Only the error log showed gocql: no response received from cassandra within timeout period. A restart then crash-looped on unable to create session until Cassandra came back.

Fix. A triage order, written down and reused on the recurrence two weeks later: count gocql errors in the error log since the last boot line; read the hourly synthetic logins as a success monitor, where a gap in cadence is the outage window; only then open the browser. The recurrence was blamed on a feed deploy. Checking the deployment that was actually serving, not the branch head, showed that deploy had never gone live.

Rule. A liveness check that only asks "is the process up?" will report a wedged dependency as healthy. Silence in a structured log is a signal.

4. The network outage that was memory

Symptom. cloudflared reported no recent network activity on all four edge connections and DNS lookups timing out. Production login returned 530, then on another night a 1033. Chat, match and upload all logged connection errors in the same window.

First theory. Cloudflare. DNS. The tunnel.

Evidence. Four independent edge IPs failing at the same instant means the common factor is local. journalctl -k showed the OOM killer. The box carries a standing load of roughly half its memory (Cassandra alone is about 7.6 GB) and two concurrent builds tipped it. A box that is thrashing cannot service UDP keepalives or resolve a name, so it presents as a network failure.

Fix. Serialise heavy builds across sessions, cap Node heaps, run batch jobs under systemd-run scopes with hard memory limits, and protect the datastore and auth gateway with OOM score adjustments so a batch job is reaped first.

Rule. When several unrelated services fail at once, check free and the kernel log before the network.

5. The index rewritten under a live reader

Symptom. After an index rebuild grew from 457 M to 781 M fingerprint pairs, the running recognition API answered with nonsense until restarted.

Evidence. The API memory-maps the index arrays at startup and records their lengths and bucket offsets. The builder truncated and rewrote the same inode. The running process kept its old offsets over new bytes. Separately, a single query expanded to about 9 GB of candidate rows and the OOM killer took the API under its 3 GB cap.

Fix. finalise_index() writes .tmp siblings and renames them into place, metadata last. A rename keeps the old inode alive for the running reader and the new files appear whole. Queries are now looked up in 4,096-pair slices, which bounds the expansion to roughly 4 M rows.

Rule. Never mutate a file another process has mapped. Publish by rename.

6. The upload Blob that sends nothing

Symptom. One track of an album upload never sent a byte. Twice. Deterministic. Worker logs showed the CORS preflight succeed and then no PUT.

Evidence. Two stacked faults. The part upload was a bare fetch with no deadline, so the hung request never errored, never retried, and held its saga lane for fifty minutes. Underneath it: every checkpoint re-put the whole IndexedDB row, including the file Blobs. A Blob the saga had already read went stale, and a stale Blob handed to XHR sends nothing and fires no error.

Fix. An XHR with a 45 s idle timeout on progress, plus an opt-in stall watchdog in the shared saga runtime. Storage v2 keeps files in their own store, written once and never rewritten. Each part is read into memory under a deadline before sending, and an unreadable file makes the saga ask for it again.

Rule. Debug "upload hangs" by looking for a preflight with no request after it. Never rewrite Blob-holding rows on a hot path.

7. The gate that passed because it stopped looking

Symptom. None. That was the problem.

Evidence. The site's post-build gate scans for redaction leaks, broken links and external requests. It read out/. The site stopped writing out/ when it moved from static export to Workers, but the stale export stayed on disk and satisfied the gate's only sanity check, "does the directory exist". For three days and about forty commits it reported the same counts and PASS without reading a byte of the shipping site.

Fix. Repointing alone would leave the same hole one directory over. The gate now reconciles the files it finds against the build's own prerender manifest, in both directions, so pointing it at the wrong tree can no longer produce a PASS. A companion script plants a leak, a broken link, an external resource and a stray file and asserts the gate refuses each.

Rule. A missing gate gets noticed. A gate that passes because it stopped looking gets trusted. Make every gate cross-check its input against something the build emits, and watch it fail at least once.

Two more, briefly

• A Worker that hung forever on Postgres. An unreachable Hyperdrive left connect() pending until the runtime cancelled the request with an error carrying no CORS headers, which a browser renders as "Failed to fetch". Every Postgres client now has a kill-switch guard, an 'error' listener and a timeout, so a repeat degrades to a 503 that names its cause. Also learned: check for the configuration variable, never the comment block warning you not to remove it.
• A benchmark ceiling that belonged to the harness. A runtime comparison reported "80 rps, Postgres-bound" for two contenders. The real ceiling was the fault-injection proxy, pinned at 100% of its half-CPU budget. Every run was invalidated and repeated. Check the measurement path's own CPU before you believe any ceiling.

Honest gaps

• No paging. Most of these were found by a person reading logs. There is no alerting on consumer lag, structured-log silence or OOM events yet.
• The reuse-policy change in incident 2 is not shipped. The frontend degrades cleanly; the backend still revokes on a late replay.
• The memory ceiling is mitigated, not solved. The box is still shared by production datastores and development work.
• Incident 4 has one unexplained reboot on record whose cause was never established.