32 · Global Feed Gateway: One Feed Every Product Mode Plugs Into

The shape

 iOS (Swift package)        web (app/globalfeed engine)
          │                            │
          └──────────┬─────────────────┘
                     ▼
        Worker: one route table, one deploy
                     │
                     ▼
        projection(item, viewer, mode, surface)   <- every read goes through here
                     │
       ┌─────────────┼───────────────┬──────────────┐
       ▼             ▼               ▼              ▼
      D1        R2 (private)    Queue consumer   mode adapters
  items, graph   media bytes    scan, classify   resolve at read time

One Worker, deployed once, serving fetch, queue and scheduled. Measured on the working tree: 12,408 lines of TypeScript outside tests, 4,670 lines of tests, and 244 tests, 244 passing under plain node --test. The generated OpenAPI document covers 76 operations and the AsyncAPI document 13 messages, and both have a --check mode that fails when the code and the contract drift.

Why D1, not the Postgres on the origin box

Every other feed in the platform reads from a single droplet that runs Cassandra, Kafka, Postgres and a dozen services. That box has a memory ceiling and has rebooted under OOM more than once. A feed is the most read-heavy thing in the product, so the decision was that feed reads never ride the origin box. D1 sits at the edge next to the Worker. Pull-on-read instead of fan-out-on-write, because at this scale a write-time fan-out buys latency the product does not need and costs a second copy of every access decision.

That second copy is the thing this design is built to avoid.

The one idea: three axes, one projection

Every item stores three independent facts:

None is inferred from another, from a row count, or from a missing value. One function reads all three together with who is asking, in which mode, on which surface. Every list, count, search hit, story ring, notification preview, share card, bookmark list and adapter tile calls it.

The reason is in the source comment. BoxOffice had shipped seven audience leaks, and they were the same defect in seven places: a surface that read items without the check. So the rule is written once.

The rule exists in two forms, and a test proves they agree

A list query cannot load rows and filter them in JavaScript. A single row check cannot be a WHERE clause. Two copies of an access rule drift. So each clause in service/projection.ts carries both its SQL fragment and its row check, side by side. The list query gets the SQL, the permalink gets the JavaScript.

tests/projection/matrix.test.ts builds a fixture matrix of 4,320 items across 660 viewer, mode, audience and surface combinations and checks it three ways, against an oracle that imports neither form. The matrix earned its keep: it found that archiving is not a visibility value, so an archived public item stayed public, and the Archived tab would have served other people's archived items to anyone who asked. That is now an ownerOnly flag on the surface policy, with a comment naming the leak.

Two rules that follow from it

• Hidden means not found. A private item, a blocked author's item and a nonexistent id return byte-identical 404 bodies. A 403 is only for an action refused on something you can already see, because confirming that a private item exists is itself the leak.
• Nothing copies an axis. Bookmarks, notifications, adapter references and search documents store an item id and re-check at read time. An item re-rated adult falls out of a general session's saved list with no write anywhere.

Hexagonal adapters, enforced by the schema

A BoxOffice film or a Billboard track can appear in the feed. The adapter reference stores the source, the type and the native id, and nothing else. No title, no URL, no audience, no moderation state. Those are resolved from the owning service at read time, because a copied access fact goes stale and a stale one is a leak. gf_adapter_refs has no column for any of them, so the rule is enforced by the table definition rather than by code review.

The gateway side (adapter/port.ts, resolve.ts, boxoffice.ts) is built. BoxOffice's own resolve endpoint is not, which is stated in the build record.

Writing the spec first, and what it found

The specification is 14,897 lines with 1,415 identifiers and more than 1,600 acceptance criteria, plus a deployment checklist of gates. Writing it against the real code surfaced blockers before any feed code existed, each verified at a file and line:

• BoxOffice's hourly reconcile adopted every ready video asset in the shared Stream account that had no catalog row. Any other product putting video there would have leaked into the BoxOffice catalog within an hour. The feed later moved video to R2 entirely, so a feed upload can never reach that job.
• The privacy policy promised EU storage and no third-party ads, which constrained where media could live and how ads could be served.
• The staff roles CHECK constraint had no Vitals mode and no ads roles.
• Storefront had no creator subscription model, so a subscribers-only visibility value had nothing to check against.

Media: fail closed, by design

Uploads go through R2 multipart. Video is parsed by range reads, refused unless it has exactly one moov, one mdat, one H.264 track and at most two AAC tracks, then streamed back out with udta, meta and stray tracks dropped (that is where GPS and device tags live) and the index moved to the front. Verified with ffmpeg: zero decode errors and identical frame checksums.

Then a scan gate. Frames and images go through a classifier, and a known-content hash port answers for exact files and for perceptual image matches. No vendor is wired to that port, so in production nothing releases. That is deliberate. A media path that publishes without a known-content check is not a feature.

Bugs worth naming

• pageLimit() capped every unbounded list to one item when no ?limit= was passed. URLSearchParams.get returns null, and Number(null) is 0, not NaN, so the "invalid, use the default" branch never ran. Present since the first build, found while extending it, fixed with a comment explaining why.
• Comment @mentions now pass the same visibility gate as everything else. A security review asked for it, and it is deployed.
• Profile edits use If-Match with an ETag and creation uses If-None-Match: *, which is optimistic concurrency on a full-replace PUT. A client that omitted both was refused rather than silently overwriting.

The scaling finding, from someone else's benchmark

FeedBench, the runtime study, ran this contract on Workers, Go and Spring. Every feed section's predicate is mode AND audience IN (...) AND archived_at IS NULL AND (ownerBranch OR visitorBranch), and the existing partial indexes require state = 'ready' or visibility = 'public'. SQLite uses a partial index only when the query implies its WHERE. This one does not, so D1 read the whole table and sorted. At 200,000 items the suggested section took about 1.35 s median per query. Two indexes, no code change, and saturation rose by an order of magnitude on all three runtimes. The production feed has the same predicate shape. It was reported, not silently patched.

Honest gaps

• No hash vendor, so no media releases in production. Text and link posts work. The vendor contract test is ready to run against an adapter.
• No real email provider exists in the account; the email port is built.
• BoxOffice's resolve endpoint for adapters is not built.
• The web UI for Billboard's feed is on a branch, not merged.
• The tuned indexes from the benchmark have not been applied to production.
• Video has no transcoder: HEVC, VP9 and fragmented MP4 are refused by name.
• Owner overrode the spec's own "no chat" boundary, so the gateway carries a DM system that duplicates GlobalChat. That decision is recorded, with the recommended alternative, in the repo.