Measurement and its limits
XRFM reports per-domain compression metrics for Native models, internal experiment records for CoderSec and bounded quality measurements for datasets. No metric here is a proxy for general assistant capability.
What BPB measures
Bits per byte (BPB) is the average number of bits required to encode each byte of a held-out domain under a model. Lower BPB means the model assigns higher likelihood to that data. It is a language-modelling and compression measure. It does not measure programming correctness, security judgement, instruction following, factuality or robustness.
All Native BPB results are per-domain. There is no aggregate score and therefore no weighting scheme to report. Step 0 is the randomly initialized baseline; final is after the listed training steps. The 20.08M tokens in the shard dataset are unique prepared tokens, while 58.98M/10.49M are tokens processed during training.
| Domain | 25M step 0 | 25M final | 85M step 0 | 85M final |
|---|---|---|---|---|
| en | 3.4651 | 1.8152 | 3.4799 | 2.2930 |
| code | 4.0011 | 0.9966 | 4.0077 | 2.1978 |
| ur | 2.6101 | 1.3674 | 2.6492 | 1.8611 |
| ur-Latn | 3.6980 | 1.8102 | 3.7126 | 2.3555 |
| ar | 2.7293 | 1.5708 | 2.6961 | 1.9244 |
| hi | 1.8358 | 0.9211 | 1.8626 | 1.2443 |
| es | 4.0700 | 2.1618 | 4.0712 | 2.7739 |
| fr | 4.0781 | 2.1611 | 4.0820 | 2.8644 |
The 85M checkpoint saw 10.49M processed tokens versus 58.98M for 25M. Higher parameter count did not overcome the shorter training run. Treat this as a scaling probe, not a model ranking.
Native comparisons and tokenizer evidence
The tokenizer tournament compared four candidates (T1–T4) and a legacy 2K control on cross-script fertility, byte fallback and lossless round-trip. T4 (32,768 Script-Protected Byte-BPE) won, with reported fertility ratios of 0.702× for Urdu, 1.103× for Arabic and 1.047× for Hindi relative to the English reference, 0% byte fallback and 100% lossless round-trip. These are tokenizer efficiency measurements, not downstream capability scores.
CoderSec evaluation
Small suites, honest language
The base-model bake-off compared Qwen3-1.7B and Qwen2.5-Coder-1.5B-Instruct on a small internal coding/defensive-security/multilingual set. Reported averages were 0.8611 and 0.8333 respectively. Security checks include an 8-prompt weaponization screen, 7 false-refusal prompts and 4 Roman-Urdu perturbation pairs. These are preliminary internal tests with very small samples.
A zero observed failure rate on 8 prompts does not prove zero real-world risk. No comprehensive safety evaluation or independent benchmark has been published, and no CoderSec checkpoint is publicly available.
Dataset validation
Native shards: 8 train files, 9,577 sequences, 19.61M train tokens, SHA-256/MinHash deduplication and a documented contamination manifest. CoderSec SFT: 24,100 rows measured; 24,100 malformed ChatML rows; 2,100 placeholder code rows; 11,993 exact duplicates; 0 empty assistant responses; 0 syntax errors in the 2,100 parsed Python blocks.
Limitations
- No held-out human evaluation or instruction-following benchmark for Native models.
- No automated code-execution correctness benchmark for CoderSec.
- Security red-team samples are too small to support population-level claims.
- Dataset quality measurement is reproducible but the corrected dataset has not been published.
- Experiment records are internal artefacts, not peer-reviewed results.