Original transformer, from zero
25M and 85M parameter checkpoints trained from random initialization on a 20.08M-token multilingual and code corpus.
- Grouped-query attention, RoPE, RMSNorm, SwiGLU
- Exact-resume probe with zero weight difference
- Per-domain held-out BPB measurements