Track A · XRFM Native

XRFM Native 85M

An 86.4M-parameter scaling probe trained from random initialization on the same native shards. It processed fewer tokens than the 25M run and has higher final BPB.

Model lineage

Random initialization; no third-party pretrained weights. The documented revision is dbe5f102. Training used config/native-85m.yaml and the same dataset revision as Native 25M.

Architecture

  • d_model=640, 14 layers, 10 query heads, 2 KV heads
  • d_ff=1920, context 2,048, RoPE, RMSNorm, SwiGLU
  • 86,446,464 tied parameters

Training

160 steps over 10,485,760 processed tokens on Tesla T4 hardware; 994.72 s wall time and 2,580.41 MB peak VRAM. The exact-resume probe reported a maximum weight difference of 0.0 and final training loss of 6.8362.

This was a shorter scaling experiment. More parameters alone did not produce better held-out BPB than the 25M checkpoint, which saw more data. Do not describe 85M as a stronger assistant.

Use and limitations

Suitable for scaling comparisons and reproducibility studies. Not suitable for production chat, coding, security advice or high-stakes use. It is not instruction-tuned.

Held-out BPB: random initialization → step 160

DomainStep 0FinalDelta
en3.47992.2930−1.1869
code4.00772.1978−1.8099
ur2.64921.8611−0.7881
ur-Latn3.71262.3555−1.3571
ar2.69611.9244−0.7717
hi1.86261.2443−0.6183
es4.07122.7739−1.2973
fr4.08202.8644−1.2176