Evaluation

Measurement and its limits

XRFM reports per-domain compression metrics for Native models, internal experiment records for CoderSec and bounded quality measurements for datasets. No metric here is a proxy for general assistant capability.

What BPB measures

Bits per byte (BPB) is the average number of bits required to encode each byte of a held-out domain under a model. Lower BPB means the model assigns higher likelihood to that data. It is a language-modelling and compression measure. It does not measure programming correctness, security judgement, instruction following, factuality or robustness.

All Native BPB results are per-domain. There is no aggregate score and therefore no weighting scheme to report. Step 0 is the randomly initialized baseline; final is after the listed training steps. The 20.08M tokens in the shard dataset are unique prepared tokens, while 58.98M/10.49M are tokens processed during training.

Domain25M step 025M final85M step 085M final
en3.46511.81523.47992.2930
code4.00110.99664.00772.1978
ur2.61011.36742.64921.8611
ur-Latn3.69801.81023.71262.3555
ar2.72931.57082.69611.9244
hi1.83580.92111.86261.2443
es4.07002.16184.07122.7739
fr4.07812.16114.08202.8644

The 85M checkpoint saw 10.49M processed tokens versus 58.98M for 25M. Higher parameter count did not overcome the shorter training run. Treat this as a scaling probe, not a model ranking.

Native comparisons and tokenizer evidence

The tokenizer tournament compared four candidates (T1–T4) and a legacy 2K control on cross-script fertility, byte fallback and lossless round-trip. T4 (32,768 Script-Protected Byte-BPE) won, with reported fertility ratios of 0.702× for Urdu, 1.103× for Arabic and 1.047× for Hindi relative to the English reference, 0% byte fallback and 100% lossless round-trip. These are tokenizer efficiency measurements, not downstream capability scores.

CoderSec evaluation

Security shield artifact representing the CoderSec evaluation boundary
Limited-sample assessment

Small suites, honest language

The base-model bake-off compared Qwen3-1.7B and Qwen2.5-Coder-1.5B-Instruct on a small internal coding/defensive-security/multilingual set. Reported averages were 0.8611 and 0.8333 respectively. Security checks include an 8-prompt weaponization screen, 7 false-refusal prompts and 4 Roman-Urdu perturbation pairs. These are preliminary internal tests with very small samples.

A zero observed failure rate on 8 prompts does not prove zero real-world risk. No comprehensive safety evaluation or independent benchmark has been published, and no CoderSec checkpoint is publicly available.

Dataset validation

Native shards: 8 train files, 9,577 sequences, 19.61M train tokens, SHA-256/MinHash deduplication and a documented contamination manifest. CoderSec SFT: 24,100 rows measured; 24,100 malformed ChatML rows; 2,100 placeholder code rows; 11,993 exact duplicates; 0 empty assistant responses; 0 syntax errors in the 2,100 parsed Python blocks.

Limitations

  • No held-out human evaluation or instruction-following benchmark for Native models.
  • No automated code-execution correctness benchmark for CoderSec.
  • Security red-team samples are too small to support population-level claims.
  • Dataset quality measurement is reproducible but the corrected dataset has not been published.
  • Experiment records are internal artefacts, not peer-reviewed results.