Language models / Training experiment
LLM Maker
Train small language models on one GPU, with resumable runs and an inspectable training history.
- My contribution
- I built the data preparation, tokenizer, training and instruction-tuning pipeline described in the research write-up.
- Tools & methods
- PyTorch · Tokenizers · Pretraining · Instruction tuning
The problem
What this project investigates
Long training runs on a single machine need to survive interruptions. Small models also need evaluation that distinguishes lower training loss from useful answering behaviour.
The build
What I built
The project documents deterministic data selection, tokenizer training, pretraining, masked-loss instruction tuning and local chat. Checkpoint resume and pause controls support multi-hour runs.
Technical choices
Decisions behind the build
Design for interrupted runs
Checkpointing and resume behaviour are part of the training infrastructure.
Compare like with like
The write-up limits perplexity comparisons to compatible tokenizers and distinguishes pretraining loss from instruction-response loss.
Current status
What the evidence shows
The later research page reports completed training and includes a recorded generation log. An older note still labels a run paused. The recording and trainer replay do not perform live inference in this prototype.
Further work & evidence gaps
Task-level factuality and instruction-following evaluations are listed as ongoing work; in-browser inference is planned.