AgentNet is the first large-scale desktop computer-based intelligent agent trajectory dataset released in 2025 by the XLANG Laboratory of the University of Hong Kong, in collaboration with Moonshot AI, Stanford University and other institutions. The related paper results are "OPENCUA: Open Foundations for Computer-Use Agents", which aims to support and evaluate cross-platform GUI operation agents and vision-language-action (VLA) models. This dataset contains 22.6K manually annotated computer usage task traces, covering Windows, macOS, and Ubuntu, and over 200 applications and websites. The scenarios fall into four categories: office, professional, daily, and system. It is suitable for training and evaluating desktop automation, multi-application processes, and cross-platform agents.

Data structures and fields

Each sample contains:

• Task metadata: task number (task_id), instruction (instruction);
• Quality rating: completion, consistency, efficiency, and difficulty;
• Summary description: natural_language_task, actual_task;
• Trajectory array: traj (operation steps recorded in chronological order).

Trajectory steps (traj)structure:

• Each step contains index, image (screenshot), and value objects:

• observation (scene observation), thought (thinking/planning), action (natural language action), code (executable code, such as PyAutoGUI), last_step_correct, last_step_redundant, and reflection.

  

Citation

@misc{wang2025opencuaopenfoundationscomputeruse,
title={OpenCUA: Open Foundations for Computer-Use Agents},
author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Haotian Yao and Ziwei Chen and Qizheng Gu and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu},
year={2025},
eprint={2508.09123},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2508.09123},
}