3 — Match Service: Geo-Partitioned Distributed Ranking
Rank 3 of 28 · Tier S · app/MatchServiceStack: Go, Cassandra / ScyllaDB (gocql + gocqlx), Redis + Redis Streams, Kafka (segmentio/kafka-go), WebSocket, Zap
Status: Running. Two production distributed-systems defects found, fixed, and documented in-source.
Why this ranks here. This project contains the two best stories in the
entire tree. Both are textbook distributed-systems failures that only appear
under real load, in real clusters, with real consumer groups — the kind an
interviewer cannot tell you were rehearsed, because the fix is committed with
the reasoning above it.
Geo-partitioned ranking, and where it went wrong
Ranking loops wrote score updates using the region they were iterating rather than the region the card actually lived in. Cassandra UPDATE is an upsert, so each wrong write created a phantom partition instead of failing, nameless cards leaked into real decks. Separately, a Redis consumer group had 952 unacknowledged messages and no reclaim path.
What it is
The candidate-generation and ranking pipeline: a geo-partitioned Cassandra
table of match cards, ranking pipelines that write scores back, a "sorting hat"
that indexes ranked cards into discovery decks, a fan-out worker, a Redis
Streams feed index, and a WebSocket hub for realtime match delivery.
dating_global.match_cards is partitioned by (service_region, geohash4).
The ranking pipelines write scores back to it — but they iterate a configured
list of regions, and the region in a pipeline's loop is the pool being
ranked, not necessarily the pool the user's card actually lives in.
Cassandra UPDATE is an upsert. So writing with the loop region did not update
a row — it silently created a partition: one phantom match_cards row per
user per configured region, carrying only score columns and a null
display_name.
The sorting hat then indexed those phantoms into the discovery tables, putting
nameless cards into real user decks.
The fix is a partition-lookup layer: resolve the user's true partition before
writing, never infer it from the iteration context.
Why this is a great story: it is the single most common Cassandra footgun
(upsert-on-write with a wrong partition key), it produced a visible product
symptom, and the root cause is three inference steps away from that symptom.
War story #2 — the leaking pending-entries list
stream_reclaim.go
A Redis consumer group moves a message into the consumer's pending-entries
list (PEL) on delivery and removes it only on XACK. Both feed consumers
continue past a processing failure without acknowledging — which is correct;
acknowledging a message you failed to process throws the data away.
But nothing ever came back for those entries. They accumulated. At the time of
the fix, feed_index:writes held 952 pending messages, the oldest long
past any useful age.
The fix is an explicit reclaim path: XAUTOCLAIM-style recovery of aged
pending entries, so failed work is retried rather than orphaned.
Why this is a great story: it demonstrates you understand that
at-least-once delivery is not free — somebody has to own the retry, and if
nobody does, the queue silently becomes a graveyard.
Other substance
match_pair_gate.go + test — the mutual-match invariant, gated so a pair
cannot be created twice from both sides concurrently.
conversation_id.go + test — deterministic conversation identity so two
services independently derive the same id.
exclusion.go + test — seen/blocked exclusion sets at candidate-generation time.