Track A · XRFM Native

XRFM Native 25M

An original 25.2M-parameter decoder-only transformer trained from random initialization. Not a production coding assistant.

Model lineage

XRFM Native 25M was initialized randomly and trained on xrfm-native-shards-v1. No third-party pretrained weights were loaded. The checkpoint revision documented here is 3487b9c8.

Architecture

  • Decoder-only transformer, XRFMModel
  • Grouped-query attention: 6 query heads, 2 KV heads
  • RoPE, RMSNorm, SwiGLU
  • d_model=384, 8 layers, d_ff=1024, context 2,048
  • 25,200,000 tied parameters

Training

900 steps over 58,982,400 processed tokens on Tesla T4 hardware. The dataset contains 20,080,975 unique prepared tokens; the larger figure reflects repeated passes. Wall time was 1,572.03 s with peak VRAM 3,865.93 MB. Exact-resume testing produced a maximum weight difference of 0.0.

Tokenizer and languages

The T4 Script-Protected Byte-Level BPE tokenizer has a 32,768 vocabulary and was selected in a four-way tournament. Evaluation domains are English, code, Urdu, Roman Urdu, Arabic, Hindi, Spanish and French.

Intended use and limits

Use for architecture research, tokenizer studies and reproducibility experiments. Out of scope: production chat, coding agents, security advice and high-stakes decisions. The model is not instruction-tuned and can generate incorrect or biased text.

Held-out BPB: random initialization → step 900

DomainStep 0FinalDelta
en3.46511.8152−1.6499
code4.00110.9966−3.0045
ur2.61011.3674−1.2427
ur-Latn3.69801.8102−1.8878
ar2.72931.5708−1.1585
hi1.83580.9211−0.9147
es4.07002.1618−1.9082
fr4.07812.1611−1.9170