xrfm-native-shards-v1
Revision 1652affc. Eight train shards are present and accessible; the dataset viewer shows only the 623-row validation split because .npy token shards are not rendered as tables.
| Train sequences | 9,577 × 2,049 uint16 |
|---|---|
| Train tokens | 19,613,696 |
| Validation tokens | 467,279 |
| Total prepared tokens | 20,080,975 |
| Source documents | 29,491 |
| Validation docs | 623 |
Sources and licenses
FineWeb-Edu and FineWeb-2 (ODC-By-1.0), OpenCoder Stage-2 (MIT), AI4Bharat Sangraha (CC-BY-4.0). Aggregate metadata: ODC-By. Source revisions are pinned in manifest.json.
Processing
SHA-256 and MinHash-LSH 8×4 deduplication; contamination check against configs/eval/frozen_benchmarks_v1.json; SHA-256 split with seed 42. Eight language/code validation sets: en, ur, ur-Latn, ar, hi, es, fr, code.