Eureka Bench Logs is a dataset of large-scale foundation model evaluation logs released by Microsoft in 2024. The associated research paper is "Eureka: Evaluating and Understanding Large Foundation Models", which aims to provide deep insights into and evaluate the reasoning capabilities of large foundation models in complex tasks.

The dataset contains run logs from the Eureka ML Insights framework, recording detailed performance of various model families and specific models across different benchmarks. Data is organized by benchmark, model family, and model name, including results from reasoning-time scaling studies on models such as Phi-4-reasoning. This information helps researchers analyze performance differences and optimization directions for models on tasks such as image-to-text.

Dataset Composition

• Benchmarks: Logs are categorized by specific evaluation benchmarks, covering multimodal tasks such as image-to-text.
• Model Families: Data is grouped by model family, facilitating comparison of progress among different models within the same architecture.
• Model Names: Each log entry corresponds to a specific model instance, recording detailed inference and evaluation data.

These log data provide rich empirical material for evaluating and understanding large foundation models.