ClimbLab is a high-quality pre-trained language model dataset released by NVIDIA in 2025, with related paper results available at 「CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training, aimed at providing rigorously filtered and cluster-optimized training datasets for large language models.

This dataset consists of a corpus containing 1.2 trillion tokens, divided into 20 semantic clusters.
It primarily targets research and development purposes, formatted in Parquet, with a data scale of approximately 40 billion tokens (the officially provided pre-tokenized version), collected and annotated automatically.
Based on Nemotron-CC and SmolLM-Corpus, it performs semantic restructuring and filtering using the CLIMB clustering method.
The specific process includes dividing the data into 1,000 groups based on topics, scoring them using two classifiers—ad detection and educational value assessment—and removing low-quality data.

Dataset Composition

The ClimbLab dataset's structure and composition are as follows:

• Data Format: In Parquet format, including token sequences processed via the GPT-2 tokenizer. To retrieve raw texts, use the official detokenize_climblab.py script for de-tokenization.
• Data Clustering: Divided into 20 independent clusters organized around semantic relevance, helping improve pre-training efficiency.
• Data Scale: Original corpus size was 1.2 trillion tokens; the officially published pre-tokenized version amounts to roughly 40 billion tokens.
• Content Characteristics: Filtered through ad removal and evaluated for educational merit ensuring both quality and pedagogical relevance.

Citation

@article{diao2025climb,
  author    = {Shizhe Diao and Yu Yang and Yonggan Fu and Xin Dong and Dan Su and Markus Kliegl and Zijia Chen and Peter Belcak and Yoshi Suhara and Hongxu Yin and Mostofa Patwary and Celine Lin and Jan Kautz and Pavlo Molchanov},
  title={CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training}, 
  journal   = {arXiv preprint},
  year      = {2025},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url={https://arxiv.org/abs/2504.13161}, 
}