LLM MAKER
What does “training a language model from scratch” actually involve when the datacenter is one RTX 3090 under a desk? This is the full pipeline — corpus to tokenizer to transformer to a 4-billion-token training run — with the real loss curves, the real crashes, and a replayable trainer console.
Watch it answer, token by token
A one-minute film of the finished 97.5M model: the architecture, the real pretraining curve, then four chat exchanges rendered one token at a time with the model's actual next-token distribution on screen beside them.
Nothing in the film is mocked. A capture script loads best.pt on the 3090 and generates with the same sampler the chat app uses, recording every step — chosen token, its probability, the top-5 alternatives, distribution entropy, wall-clock milliseconds. The video is a replay of that log. Exchange 2 is the honest one: asked what photosynthesis releases, the model says water — Ox led the distribution at 19.5% and sampling drew water at 10.9%. At this scale the right answer is often present and lost to the dice.
Bytes in, predictions out
“From scratch” here means no pretrained anything: no borrowed tokenizer, no downloaded weights, no fine-tuning of somebody else's model. The pipeline starts with 26.6 GiB of raw educational web text and ends with a checkpoint that can continue a sentence. Everything in between — bytes → BPE tokens → embeddings → stacked attention → a probability over the next token — is code in this repo, small enough to read in an afternoon.
“the” is just three bytes.16,384 subwords. Text becomes uint16 IDs.768-dim learned vector; RoPE rotates in position information.The single-GPU realities shape everything else. A 4B-token run is ~23 hours of computeat 47,700 tokens/sec — and this run actually died twice in earlier incarnations, once to Windows sleep and once to session teardown, both silently. The fix wasn't hope: a Scheduled Task owns the process, --resume restores model + optimizer + RNG + data cursor bit-exactly, and a STOPfile pauses the run in two seconds when the GPU is needed for something else. The finished run was deliberately paused twice mid-flight; the loss curve below doesn't show a ripple. OOM management is why the global batch is 65,536 tokens built from 8-block micro-batches — the 100M model peaks at 7.9 GiB of the 3090's 24.
Six stages, one afternoon of code
Every stage is a standalone script with deterministic outputs — the same seeds reproduce the same subset, tokenizer, and packed blocks, byte for byte.
CORPUS
FineWeb-Edu sample-10BT: 14 Parquet shards, 26.6 GiB, 9.67M documents. A keyed BLAKE2b hash ranks every document ID, so the 4B-token subset is deterministic — rerun the selector with the same seed and you get the identical split, and bigger budgets are supersets of smaller ones.
TOKENIZER
A byte-level BPE trained on a hashed sample of the corpus itself — 16,384 merges, special tokens <bos> <eos> <unk>. Byte-level means no out-of-vocabulary failures, ever. Then every document is encoded, terminated with <eos>, concatenated, and packed into 1,024-token uint16 blocks on disk.
MODEL INIT
Decoder-only transformer: 12 layers, d_model 768, 12 heads (64-dim each), SwiGLU MLPs (hidden 2,048), RoPE positions, RMSNorm, tied input/output embeddings — 97,536,768 parameters, and the trainer asserts that exact count so a config typo can't silently change the model.
TRAINING LOOP
Fused AdamW (β 0.9/0.95, weight decay 0.1), grad-clip 1.0, bf16 autocast with fp32 loss. Global batch 65,536 tokens = 8-block micro-batches × gradient accumulation. Blocks are memory-mapped and read through a seeded permutation — 1.37 s/step, ~47,700 tok/s sustained on the 3090.
CHECKPOINT / RESUME
Every 250 steps last.pt captures model, optimizer, scheduler, data cursor, and all four RNG states — resume is bit-exact (verified delta: 0.0). A Windows Scheduled Task owns the process so it survives session teardown, a STOP file pauses it gracefully in ~2s, and an auto-resume loop relaunches after any crash.
SAMPLING
After 61,036 steps the best checkpoint serves completions (top-k 40, temperature 0.75) through a tiny local Flask app. The base model continues text; a follow-up SFT pass (Dolly + SQuAD, loss masked to responses) teaches it to answer and, crucially, to stop.
Real curves from a real run
Charts drawn from runs/smoke_100m_16k_4b/metrics.jsonl — the training log of the completed 4B-token run (June 2026).
metrics.jsonl. Final train/val gap is −0.035 — the model never overfit; it ran out of budget, not data. The orange dashed lines are where the run was deliberately paused (STOP file) and later resumed from last.pt— the curve doesn't even flinch.metrics.jsonl). One continuous cosine over the whole budget — no warm restarts, because bolting a fresh cosine onto already-annealed weights is a different, worse regime. Peak is 6e-4, lower than the 12M model's 1e-3: bigger models want gentler learning rates.smoke_100m_16k.yaml(12 layers × d_model 768, SwiGLU hidden 2,048, vocab 16,384, tied embeddings). At 10M params the embedding table ate 21% of the budget; at 97.5M it's down to 12.9% — scaling up spends the new capacity on compute (MLPs + attention), not lookup.TRAINER.TTY
Replay the run yourself: 61,036 optimizer steps compressed into about fifteen seconds, following the real loss curve — including both mid-run pauses. When it finishes, ask the checkpoint for a sample.
Model registry & status
Four pretraining runs and two SFT passes so far. Perplexity is only comparable within the same tokenizer — the 12M and 100M models share the 16k tokenizer, so their numbers are a fair scaling comparison.
| Run | Params | Vocab | Train tokens | Val loss (ppl) | Note |
|---|---|---|---|---|---|
| smoke_10m_full_200m ↗ | 10.1M | 8,192 | 200M | 3.559 (35.1) | the original baseline |
| smoke_12m_16k_4ep ↗ | 12.2M | 16,384 | 2.88B (4 epochs) | 3.433 (31.0) | epoch taper confirmed ~4× is the repeat ceiling |
| sft_12m_16k ↗ | 12.2M | 16,384 | 10.8k pairs | resp. 2.758 | learned to answer and stop; extractive QA works |
| smoke_100m_16k_4b ↗ | 97.5M | 16,384 | 4.00B | 2.889 (18.0) | this page's run — beat the 12M model's final loss at 52% trained |
| sft_100m_16k ↗ | 97.5M | 16,384 | 10.8k pairs | resp. 2.186 | SFT of the 100M base, same recipe as the 12M |
- Full pipeline — shard download, deterministic subset selection, tokenizer training, block packing, pretraining, SFT, local chat app
- 100M pretrain complete — 4B tokens, val loss 2.889, no overfit (gap −0.035)
- Resilience infra — bit-exact resume, STOP-file pause, Scheduled-Task auto-resume loop (survived two real crashes on the 12M run)
- SFT recipe — masked-loss instruction tuning; the model answers, stops at
<eos>, and does genuine extractive QA
- Hugging Face release— five model repos packaged locally as a “FineWeb Educational Models” collection; upload pending
- Evals beyond perplexity — held-out loss says nothing about factuality or instruction-following; building small task probes
- Live in-browser inference— replace this page's replay with the exported weights running locally in your tab
- RAG over the SFT model— a capacity-limited model can't store facts, but it can extract them from a provided passage; retrieval is the highest-value lever
- Context extension & distillation — 1,024 tokens is the trained ceiling; both need dedicated experiments