Installation
git clone https://github.com/ahmadrrrtx/xrf-foundation-model.git
cd xr-foundation-model
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev,hub]'
The import name remains xrfm. Do not install the unrelated PyPI distribution named xrfm.
Quick start
import xrfm
cfg = xrfm.load_config("config/tiny.yaml")
model = xrfm.XRFM(cfg)
tok = xrfm.BPETokenizer.pretrained()
ids = tok.encode("Assalam o Alaikum — hello in Urdu.")
print(tok.decode(ids))
Tokenizer usage
The T4 tokenizer file is packaged with the library and published with both Native checkpoints. Special tokens are atomic. The legacy vocab.json is retained only as the tournament control.
from xrfm.tokenization import HFTokenizerAdapter
tok = HFTokenizerAdapter("src/xrfm/tokenization/tokenizer_32k_multilingual.json")
ids = tok.encode("Roman Urdu: aap kaise hain?")
assert tok.decode(ids) == "Roman Urdu: aap kaise hain?"
Training configuration
Active configs: config/tiny.yaml, config/native-25m.yaml, config/native-85m.yaml. Dataset sources are pinned in configs/data/. Run the CPU smoke test before any GPU experiment:
python scripts/ci_smoke_test.py
python scripts/train_native_from_hub.py --config config/native-25m.yaml
Inference
from xrfm import generate
from xrfm.inference import GenerationEngine
engine = GenerationEngine(model, tokenizer=tok)
result = engine.generate("Once upon a time", max_new_tokens=64)
print(result.text)
Native checkpoints are small research models, not chat models. Prompt formats are plain text unless a separate experiment specifies otherwise.
Dataset preparation
Native shards are uint16 arrays of 2,049 values: 2,048 tokens plus the next-token label. Validation JSONL files are per language. CoderSec SFT is JSONL with row-level provenance and license fields.
python scripts/build_and_publish_native_shards.py --help
python scripts/filter_codersec_sft.py --help
Evaluation
python scripts/evaluate_checkpoint.py --help
python scripts/run_eval_protocol.py --help
Report BPB per domain. Do not aggregate without documenting weights. Explain that BPB is a likelihood measure.
Reproducibility
python scripts/verify_reproducibility.py
This checks pinned experiment artefacts including the tokenizer tournament, shard manifest and SFT manifest. Model and dataset revisions are recorded in the release reports.
Contributing
Read the contribution guide. Keep Native and CoderSec lineage separate, preserve tokenizer research, add tests for behaviour changes and avoid committing large artefacts.
Full documentation in the repository →