1 — Billboard Activity Graph

The problem

Playlists, follows, feed, and posts are one social graph. The naive build gives each of them its own event pipeline, its own fan-out, and its own consistency story — four systems that drift apart. The directive was one activity log and one materializer.

Complication: the service that writes activity, billboard-users, is a Cloudflare Worker. Workers cannot open raw TCP sockets — not with a library, not with a shim. The Kafka wire protocol and CQL are both permanently unreachable from a Worker.

The architecture

billboard-users (Cloudflare Worker)
      │  writes billboard_feed_events in the SAME transaction as the post
      ▼
Postgres billboard_feed_events                    ← the activity log / outbox
      │  relay: claim → produce → stamp published_at
      ▼
Kafka  billboard.activity.v1   (6 partitions, keyed by actor)
      │
      ├─ group feed-materializer   → Cassandra billboard_feed.feed_timeline
      ├─ group notification-worker → Postgres billboard_notifications
      └─ group fanout              → realtime delivery

Four processes in one binary because they share a database pool and a Cassandra session — not because they are coupled. Each consumer owns its group, so splitting them across hosts later is a deployment change, not a rewrite.

The four decisions worth defending

1. Outbox, not direct produce

A Worker cannot enlist Kafka in a Postgres transaction. Producing directly leaves a window where either a post exists with no event, or an event survives a rolled-back post. Writing the event inside the post's transaction and relaying it afterwards closes that window.

2. At-least-once, with the cost paid explicitly at every sink

The relay produces before stamping published_at. A crash between the two re-delivers. That is deliberate: a lost event cannot be recovered, a duplicate can be absorbed. Each consumer absorbs it differently:

The idempotency key is the outbox row id — stable across re-delivery by construction.

3. Three groups, not three stages

A Cassandra outage must not stop notifications. A notification failure must not keep posts out of feeds. A single shared pipeline does exactly that. Three independent groups off one topic means each can fail, lag, and retry alone.

4. Thin events, not denormalized ones

billboard_feed_events carries ~40 denormalized display columns for the activity-feed UI. None of them are in the Kafka event. A consumer that needs the actor's avatar joins for it. Baking display fields into the event freezes every one of them at write time — and they are wrong the moment a user renames themselves.

Related design directives implemented

• Fractional / LexoRank ordering, not integer positions — reordering one track must not rewrite every row after it.
• System playlists are fixed-ID playlists. "Liked Songs" is a row with a reserved id, not a parallel table with duplicate code paths.
• One follows table discriminated by target_type, not one per followable kind.
• Hybrid fan-out — push below ~10k–50k followers, pull above. The threshold is a tuning constant, not a fork in the architecture.

What was verified before building

Directive 6 ("feed materialization is a Kafka consumer writing to Cassandra") required a Kafka and a Cassandra reachable from this host. Both were confirmed present before a line was written. Directive 2 was identified as a migration, not a new column — /v1/playlists/{id}/shuffle already rewrites an integer position holding live data.

Where a directive would have required infrastructure that was not present, it was written up as blocked rather than quietly substituted.

Scope

Interview surface this opens

• Exactly-once vs at-least-once, and why you never got to claim the former
• Outbox pattern vs CDC vs dual-write, and what each costs
• Partition key choice (actor id) and the hot-partition risk it carries
• Consumer group rebalancing and lag monitoring
• Push/pull fan-out crossover and where the celebrity problem bites
• LexoRank and why integer positions do not survive reordering at scale