19 — Solomon: The Transformer, Built From the Paper

Solomon 1 of 6 · Solomon/solomon/model Stack: PyTorch only — no HuggingFace, no transformers, no pretrained anything Reference: The Annotated Transformer (Harvard NLP) — Vaswani et al. in pure PyTorch

Where this sits. The deepest "do you actually understand it" artifact in the tree. Every component is either verbatim from the reference or has its deviation justified in writing, component by component, in PLAN.md.

The component map — written before the code

Reference component	Status	Note
`attention(q,k,v,mask,dropout)`	verbatim	scaled dot-product, `masked_fill(mask==0, -1e9)`, softmax, dropout
`MultiHeadedAttention`	verbatim	4 linears, view/transpose to (b,h,t,d_k), self-attention only
`PositionwiseFeedForward`	verbatim	`w_2(dropout(relu(w_1(x))))`
`LayerNorm`	verbatim	`a_2`, `b_2`, eps 1e-6
`SublayerConnection`	verbatim	`x + dropout(sublayer(norm(x)))`
`Embeddings`	verbatim	`lut(x) * sqrt(d_model)`
`PositionalEncoding`	verbatim	sinusoidal, `div_term = exp(arange(0,d,2) · −ln(10000)/d)`
`subsequent_mask`	verbatim	`triu(ones, k=1) == 0`
`Generator`	verbatim	linear + log_softmax
`NoamOpt`	verbatim	`factor · d_model^−0.5 · min(step^−0.5, step·warmup^−1.5)`
`LabelSmoothing`	verbatim	KLDivLoss, confidence/(V−2), padding zeroed
Xavier init	verbatim	`xavier_uniform_` on dim > 1
`Encoder`, `EncoderLayer`, src-attn	dropped	no encoder in a causal LM
`EncoderDecoder`	adapted → `SolomonLM`	embed → N decoder layers (causal mask) → final LayerNorm → Generator
`Batch`, `run_epoch`, `SimpleLossCompute`	adapted	LM batches: `y = x` shifted by 1; `ntokens` excludes pad
`greedy_decode`	adapted	+ beam, temperature/top-k/top-p — decode-time only, no architecture change

The three deviations, each defended

Decoder-only. Required by the mission: a generative LM doing next-token prediction has no source sequence to encode.
Weight tying (embedding ↔ generator). Saves ~2.1M of ~7M params; on a small-corpus CPU model the regularization is worth more than the capacity (Press & Wolf 2017). The reference itself notes shared embeddings as an option.
Pre-norm — not actually a deviation. The reference code already applies norm before each sublayer ("for code simplicity the norm is first"). Saying so, rather than claiming credit for it, is the point.

Configuration

vocab 8192 · d_model 256 · N = 6 layers · h = 8 heads · d_ff 1024 · seq 256 · dropout 0.1 · tied weights → 6.84M parameters.

Sized deliberately: it trains to a real checkpoint on 16 CPU cores in hours, and the loop fits the ~13 GB of RAM left over after the production services on the same host take theirs. The model was sized by the machine's spare capacity, and that constraint is documented rather than hidden.

Decoding

Greedy, beam, and temperature / top-k / top-p — all decode-time, no architectural change. KV-cached decode was proven exactly equal to full recompute, not assumed. That proof is the thing to mention: caching is where generation quietly goes wrong, and "exactly equal" is a testable claim.

The tests that prove understanding

Causal-mask leak test — does information from position t+1 reach t?
Positional-encoding formula test — the actual sinusoid, not a shape check.
Overfit-one-batch — the canonical "is my training loop wired at all" test.
Greedy determinism — same input, same output, every time.

Interview surface this opens

Attention at the tensor level: shapes, masking, why −1e9 and not −inf
Why pre-norm trains more stably than post-norm
Weight tying: the parameter saving vs the representational cost
KV caching and how you prove equivalence rather than assert it
Sinusoidal vs learned vs rotary positional encoding