Documentation

Install, run and reproduce

Python 3.10–3.12 are tested in CI. Install from source: the PyPI name xrfm belongs to an unrelated project.

Installation

git clone https://github.com/ahmadrrrtx/xrf-foundation-model.git
cd xr-foundation-model
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev,hub]'

The import name remains xrfm. Do not install the unrelated PyPI distribution named xrfm.

Quick start

import xrfm

cfg = xrfm.load_config("config/tiny.yaml")
model = xrfm.XRFM(cfg)
tok = xrfm.BPETokenizer.pretrained()
ids = tok.encode("Assalam o Alaikum — hello in Urdu.")
print(tok.decode(ids))

Tokenizer usage

The T4 tokenizer file is packaged with the library and published with both Native checkpoints. Special tokens are atomic. The legacy vocab.json is retained only as the tournament control.

from xrfm.tokenization import HFTokenizerAdapter

tok = HFTokenizerAdapter("src/xrfm/tokenization/tokenizer_32k_multilingual.json")
ids = tok.encode("Roman Urdu: aap kaise hain?")
assert tok.decode(ids) == "Roman Urdu: aap kaise hain?"

Training configuration

Active configs: config/tiny.yaml, config/native-25m.yaml, config/native-85m.yaml. Dataset sources are pinned in configs/data/. Run the CPU smoke test before any GPU experiment:

python scripts/ci_smoke_test.py
python scripts/train_native_from_hub.py --config config/native-25m.yaml

Inference

from xrfm import generate
from xrfm.inference import GenerationEngine

engine = GenerationEngine(model, tokenizer=tok)
result = engine.generate("Once upon a time", max_new_tokens=64)
print(result.text)

Native checkpoints are small research models, not chat models. Prompt formats are plain text unless a separate experiment specifies otherwise.

Dataset preparation

Native shards are uint16 arrays of 2,049 values: 2,048 tokens plus the next-token label. Validation JSONL files are per language. CoderSec SFT is JSONL with row-level provenance and license fields.

python scripts/build_and_publish_native_shards.py --help
python scripts/filter_codersec_sft.py --help

Evaluation

python scripts/evaluate_checkpoint.py --help
python scripts/run_eval_protocol.py --help

Report BPB per domain. Do not aggregate without documenting weights. Explain that BPB is a likelihood measure.

Reproducibility

python scripts/verify_reproducibility.py

This checks pinned experiment artefacts including the tokenizer tournament, shard manifest and SFT manifest. Model and dataset revisions are recorded in the release reports.

Contributing

Read the contribution guide. Keep Native and CoderSec lineage separate, preserve tokenizer research, add tests for behaviour changes and avoid committing large artefacts.

Full documentation in the repository →