TaskBench is a dataset released by Microsoft Research in 2023, focusing on the evaluation of task automation for Large Language Models. The associated paper is available at TaskBench: Benchmarking Large Language Models for Task Automation, aiming to address challenges in data collection and evaluation for complex tasks such as task decomposition, tool invocation, and parameter prediction.

The dataset contains 28,271 high-quality samples, covering three domains: Tools, Multimedia Tools, and Daily Life APIs. The data uses Tool Graphs to represent decomposed tasks within user intents. Through the Back-Instruct method, it simulates user instructions and annotations, supporting various task decomposition structures such as nodes, chains, and DAGs, thereby providing a comprehensive benchmark for evaluating the automation capabilities of LLMs in real-world scenarios.

Dataset Composition

The data scale for each subset is as follows:

• Tools: 12,217 samples in total, with 7,458 retained after manual verification.
• Multimedia Tools: 8,904 samples in total, with 5,555 retained after manual verification.
• Daily Life APIs: 7,150 samples in total, with 4,318 retained after manual verification.

Each subset directory contains the following four files:

• data.json: Stores dataset samples.
• graph_desc.json: Stores tool graph descriptions.
• user_requests.json: Stores user requests.
• tool_desc.json: Stores tool descriptions.

Citation

@article{shen2023taskbench,
      title   = {TaskBench: Benchmarking Large Language Models for Task Automation},
      author  = {Shen, Yongliang and Song, Kaitao and Tan, Xu and Zhang, Wenqi and Ren, Kan and Yuan, Siyu and Lu, Weiming and Li, Dongsheng and Zhuang, Yueting},
      journal = {arXiv preprint arXiv:2311.18760},
      year    = {2023}
    }