Interview Stories

1. The phantom Cassandra partition

Situation. Users were seeing nameless cards in their discovery deck — real rows, no display name.

Action. Traced it back through the sorting hat to the ranking pipelines. match_cards is partitioned by (service_region, geohash4). The pipelines iterate a configured list of regions, and the region in the loop is the pool being ranked — not necessarily the pool the user's card lives in. Because Cassandra UPDATE is an upsert, writing with the loop's region did not fail and did not update: it created a new partition, one per user per configured region, holding only score columns. The sorting hat then indexed those into real decks.

Result. Added a partition-lookup layer that resolves the user's true partition before any write, so partition identity is never inferred from iteration context. Fix and reasoning committed in user_partition.go.

Follow-ups to expect

• Why didn't this error? — Cassandra has no "row must exist" write. Upsert is the only write.
• How would you catch it earlier? — a read-back assertion in the write path in non-prod, or a partition-count invariant per user in monitoring.
• Would a different partition key have avoided it? — only by making the key derivable from the row itself; the real fix is not letting the caller supply a key it does not own.

2. The 952 orphaned messages

Situation. A Redis Streams feed index was silently falling behind.

Action. A Redis consumer group moves a message to the consumer's pending-entries list on delivery and removes it only on XACK. Both feed consumers continue past a processing failure without acking — which is correct, since acking work you failed to do destroys it. But nothing ever came back for those entries. feed_index:writes had accumulated 952 pending messages, the oldest long past useful.

Result. Implemented an aged-entry reclaim path so failed work is retried rather than orphaned. stream_reclaim.go.

Follow-ups to expect

• Kafka vs Redis Streams here? — Kafka's group offsets make this failure look like lag (visible); Redis's PEL makes it look like nothing (invisible). That difference is the whole story.
• When do you give up on a message? — you need a delivery-count threshold and a dead-letter destination; unbounded retry is the same bug wearing a different hat.
• How do you alert on it? — PEL depth and oldest-pending age, both as gauges.

3. The Worker that could not speak Kafka

Situation. The directive was "feed materialization is a Kafka consumer writing to Cassandra." The service producing the activity is a Cloudflare Worker.

Action. Established first that Workers cannot open raw TCP sockets — so both the Kafka wire protocol and CQL are permanently unreachable, not merely awkward. That ruled out producing from the Worker at all. It also ruled out a dual write: a Worker cannot enlist Kafka in a Postgres transaction, so producing directly leaves a window where a post exists with no event, or an event survives a rolled-back post.

Result. A transactional outbox: the event row is written inside the same transaction as the post; a host-resident Go relay claims, produces, and stamps published_at. The relay produces before stamping — a crash between the two re-delivers, which is deliberate, because a lost event cannot be recovered and a duplicate can be absorbed. Each of the three consumers absorbs it with a named mechanism.

Follow-ups to expect

• Why not CDC / Debezium? — a legitimate alternative; it trades an application-owned relay for an infrastructure dependency and a schema-coupled connector. Say that, don't dodge it.
• Exactly-once? — no, and you should say so flatly. At-least-once plus idempotent sinks. Effectively-once at the sink is the honest phrasing.
• What's the idempotency key? — the outbox row id, stable across re-delivery by construction.

4. Three groups, not three stages

Situation. Three things must happen when a user posts: it enters followers' feeds, it generates notifications, and it fans out to realtime listeners.

Action. The tempting build is one pipeline doing all three in sequence. That means a Cassandra outage stops notifications, and a notification failure keeps posts out of feeds. Instead: one topic, three independent consumer groups, each reading every event, each failing and retrying alone.

Result. All four processes run in one binary today because they share a database pool and a Cassandra session — not because they are coupled. Splitting them across hosts is a deployment change, not a rewrite.

Follow-ups to expect

• Why not three topics? — one event, three readers. Three topics means three producers to keep consistent.
• Ordering guarantees? — per-partition, keyed by actor, so one actor's events stay ordered relative to each other. Cross-actor ordering is not promised and is not needed.
• Hot partition risk? — yes, a very high-volume actor concentrates on one partition. Mitigation is a composite key or a spill partition for outliers.

5. The event that deliberately carries almost nothing

Situation. The outbox table carries ~40 denormalized display columns for the activity-feed UI. The obvious move is to put them in the Kafka event so consumers don't have to join.

Action. Kept the event minimal — id, type, actor, target type, target id, timestamp. Consumers that need the actor's avatar join for it.

Result. Display fields are not frozen at write time. A user renaming themselves does not leave a trail of events carrying their old name forever.

Follow-ups to expect

• Isn't the join expensive at fan-out time? — yes, and it's cached. The alternative is unbounded incorrectness, which has no cache.
• When would you denormalize? — when the value is genuinely immutable at event time (the amount of a transaction, the version of a document), not when it's a mutable profile attribute.

6. The gate that stops a model shipping

Situation. A trained checkpoint is not a deployable model.

Action. Built promotion as a 9-gate evaluation — perplexity, BPC, token accuracy, distinct-2, repetition rate, LAMBADA, HellaSwag, serving p95, and throughput — with a data card recording contamination checks against the eval suites themselves. Only when all nine pass is the checkpoint copied to serving.pt, the image rebuilt against it, and the rollout performed.

Result. All 9 green; test PPL 90.0, p95 454 ms, 116 tok/s. The image on the cluster is provably the checkpoint that passed, via a SHA256 manifest.

Follow-ups to expect

• Why include latency in a quality gate? — because a model that is accurate and too slow is not shippable, and discovering that after rollout is the expensive way.
• Contamination checking? — the corpus was checked against the eval prompt suites; otherwise the gate measures memorization and passes a bad model.
• How do you roll back? — the previous serving.pt and its manifest are retained; rollback is an image tag, not a retrain.

7. The bug that takes down the host

Situation. Auditing the WebRTC call path in a Go service.

Action. Found a concurrent map access in the call path. In Go this is not a panic you can recover — it is fatal error: concurrent map writes, which terminates the runtime. So it does not degrade one call; it drops every connection on the host.

Result. Documented as G-07 in the gap register with severity and a remediation entry, rather than being papered over.

Follow-ups to expect

• How do you fix it? — sync.Map for the read-mostly case; better, a single owning goroutine with a channel so the map is never shared at all.
• How do you find these before production? — -race in CI, and the fact that this codebase has dedicated race tests (calling_race_test.go) is the answer to "do you actually do that."
• Why is a fatal worse than a panic? — no recover, no graceful shutdown, no draining. Every in-flight request dies with it.

8. Reproducing bugs on purpose

Situation. Porting a large iOS app to Kotlin/Compose, seven pre-existing defects surfaced in the original.

Action. Reproduced them deliberately in the port and wrote them down, instead of fixing them in passing.

Result. The two platforms stay behaviorally identical during the migration, so parity testing means something. The defects get fixed once, on purpose, on both platforms, as their own change.

Follow-ups to expect

• Isn't that shipping known bugs? — for the duration of a migration, yes, and it is the cheaper error. A port that silently diverges is untestable against the original.
• What if one is a security bug? — then it is not a parity question, it comes out of the queue and gets fixed on both immediately. Have this distinction ready.

Questions to ask them

Ask these; they signal the level you're interviewing at.

1. What is your delivery guarantee at the boundary between services, and where do you pay for it?
2. When a consumer group falls behind, what page fires and who gets it?
3. How do you keep client and server contracts from drifting — is it enforced, or reviewed?
4. What is the last unrecoverable production failure, and what changed structurally afterward?
5. For a fully remote role at this level: how are design decisions made and recorded when nobody is in a room?