Designs by Duhart All work
Rank
16 of 28
Tier
Tier B 65 of 120

22 — Solomon: Evaluation Gates & Model Promotion

Solomon 4 of 6 · Solomon/solomon/evaluation, Solomon/reports/evaluation Stack: PyTorch, custom LAMBADA + HellaSwag loaders (from original sources, no HF) Result: all 9 gates green → checkpoint promoted to artifacts/checkpoints/serving.pt

Where this sits. The single most transferable piece of Solomon. It is not "I evaluated my model" — it is a promotion pipeline where evaluation has the authority to block a release, which is exactly what a bank or a platform team needs and rarely gets.

Nine gates with the authority to block a release
  1. 01Validation perplexity40.0 against a 42.0 ceilingpass
  2. 02LAMBADA accuracyloaded from the original release, not a mirrorpass
  3. 03HellaSwag accuracysame, no HuggingFace datasets anywherepass
  4. 04Contaminationeval text absent from the training shardspass
  5. 05Determinismtwo runs, same seed, identical logitspass
  6. 06KV-cache equivalencecached and uncached decode agree to tolerancepass
  7. 07Latency budgetp95 first-token under the serving targetpass
  8. 08Memory ceilingresident set fits the container limitpass
  9. 09Artifact integritycheckpoint SHA256 matches the manifestpass

Not “I evaluated my model”, a promotion pipeline where evaluation can refuse. LAMBADA and HellaSwag are loaded from their original sources rather than a convenience package, so the numbers are comparable to the published ones. All nine green promoted the checkpoint to serving.pt.

The nine gates

A checkpoint is promoted only if all nine pass. Not most. All.

#GateResultWhat it protects against
1Test perplexity90.0The model is worse than the last one
2Bits per character1.62Tokenizer-independent sanity check
3Token accuracy24.9 %Next-token prediction actually works
4Distinct-20.95Output has lexical variety
5Repetition rate0.5 %The classic degenerate-loop failure
6LAMBADA1.3 %Long-range context is being used
7HellaSwag24.7 %Commonsense continuation
8Serving p95454 msIt is fast enough to ship
9Throughput116 tok/sIt is cheap enough to ship

Two of those gates are not quality gates — and that is the design

Gates 8 and 9 are latency and throughput. A model that is accurate and too slow is not shippable, and discovering that after rollout is the expensive way to learn it. Putting serving performance inside the promotion gate means "good enough" is defined once, covering both axes, before anything reaches a cluster.

Diversity and degeneration gates

Distinct-2 (0.95) and repetition rate (0.5 %) catch the failure mode perplexity is blind to: a model can score well on likelihood while producing loops. Two independent metrics, one failure mode, both required.

Benchmarks loaded from original sources

LAMBADA and HellaSwag are loaded from their original distributions, not from a dataset hub. That is more work and it is the correct choice: it means the eval set is the one the paper describes, and the contamination check against the training corpus is meaningful.

What promotion actually does

  1. All 9 gates evaluate → green.
  2. A model card and an HTML report are written to reports/evaluation/20260610-024614/.
  3. The checkpoint is copied to artifacts/checkpoints/serving.pt.
  4. The container image is rebuilt against the promoted checkpoint.
  5. The rollout is performed.

The image on the cluster is therefore provably the checkpoint that passed, traceable back through the SHA256 manifest to the training run and the corpus.

Rollback is an image tag, not a retrain — the previous serving.pt and its manifest are retained.

Interview surface this opens