Proof-Pile-2 is a tokenized dataset of 55 billion math and science documents. It is a blend of scientific papers, math-related web content, and math code up to date as of April 2023 (excluding a specific subset of Lean proof steps). This dataset was created to train Llemma 7B and Llemma 34B models. It consists of three subsets:

• arxiv (29B tokens): RedPajama's ArXiv subset
• open-web-math (15B tokens):OpenWebMath A dataset containing many high-quality mathematical texts from the Internet.
• algebraic-stack (11B tokens): A new dataset of mathematical codes covering numerical computing, computer algebra, and formal mathematics.

Citation

@misc{azerbayev2023llemma,
title={Llemma: An Open Language Model For Mathematics},
author={Zhangir Azerbayev and Hailey Schoelkopf and Keiran Paster and Marco Dos Santos and Stephen McAleer and Albert Q. Jiang and Jia Deng and Stella Biderman and Sean Welleck},
year={2023},
eprint={2310.10631},
archivePrefix={arXiv},
primaryClass={cs.CL}
}