One-sentence Summary

Tencent Hy Frontier Team presents UI-Mate, a foundation GUI agent that combines an environment-grounded training stack with in-context demonstration learning to transform multimodal demonstrations into flexible subtask-level workflows, and introduces OSWorkerBench, a benchmark of 100 long-horizon office tasks across 41 applications; UI-Mate-27B attains 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena, and outperforms its Qwen3.6-27B base on OSWorkerBench by 17.7 strict-success points.

Key Contributions

• UI-Mate is a foundation GUI agent that integrates a scalable environment-grounded training stack with in-context demonstration learning. The training stack provides a closed-loop data engine that automates task generation, environment construction, rollout, filtering, and hierarchical capability balancing for supervised fine-tuning and online reinforcement learning across massively parallel environments via unified task-verifier bundles.
• The in-context demonstration learning mechanism converts multimodal demonstrations into flexible, subtask-level procedural workflows rather than replaying rigid trajectories. It selectively follows, skips, or adapts demonstrated steps and re-plans from the live interface to support demonstration-guided procedural generalization.
• OSWorkerBench is a benchmark of 100 long-horizon office tasks across 41 applications with instruction-only and demonstration-guided evaluation, separating 33 self-demo and 45 variant-demo resources. UI-Mate-27B attains 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena, reaches 41.0% strict success and 76.9% progress on OSWorkerBench (outperforming its Qwen3.6-27B base by 17.7 and 24.5 points), and on the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%.

Introduction

Foundation GUI agents can translate natural-language intent into multi-step desktop actions, but real-world deployment is limited by two bottlenecks. Training data are skewed toward short single-application tasks, leaving long-horizon workflows, cross-application transfer, and error recovery sparse, while concise instructions omit user-specific procedural details, causing inconsistent execution on similar requests. The authors address this with UI-Mate, an open-weight foundation GUI agent that combines environment-grounded training with in-context demonstration learning. A closed-loop data pipeline constructs executable tasks and verifiers, filters rollouts, and uses a hierarchical capability tree to rebalance coverage, while DemoCUA converts human or agent demonstrations into adaptive subtask-level workflows rather than rigid action replay. They also introduce OSWorker-Bench to evaluate long-horizon office tasks under instruction-only and demonstration-guided settings.

Dataset

The authors describe two dataset layers: a training and reinforcement-learning data pipeline, and the OSWorkerBench benchmark.

Training and RL Data Pipeline

Sources and composition

• Open-source computer-use datasets, including AgentNet and ScaleCUA, provide everyday tasks derived from real user activity.
• Atomic subtasks decomposed from failed or stalled rollouts focus on operations that agents find difficult.
• Instructions generated from real documents, spreadsheets, presentations, and static websites provide concrete entities and long-horizon multi-application workflows.
• Capability trees built from application specifications cover fine-grained operations that common workflows may skip.
• The overall distribution is deliberately biased toward everyday office use.

Environment construction

• An LLM identifies required files and generates executable setup code to create them when needed.
• Setup code uploads resources and configures the OS, applications, and task-specific state.
• Randomization varies wallpapers, desktop layouts, application settings, and sidebar positions while preserving task feasibility.
• Open-source documents, presentations, spreadsheets, images, videos, and audio are indexed and retrieved as real grounding resources.
• Synthetic files are used only when no suitable real file exists.

Rollout and filtering

• Rollout infrastructure supports Ubuntu, Windows, and macOS through a cloud virtual machine backend for parallel execution.
• Filtering has two stages:
  • A multimodal judge validates task and environment setup and rejects ambiguous, infeasible, malformed, or already satisfied trajectories.
  • Step-level outcome verification tracks independently verifiable deliverables across GUI observations and actions.
• Trajectories are retained only when every deliverable is supported by evidence.

Capability taxonomy and rebalancing

• Tasks are mapped to a shared capability tree with three levels: application, coarse capability, and fine-grained operation.
• A separate cross-application domain covers behaviors that connect applications.
• Data rebalancing uses target coverage, observed density, rollout success, and filtering rejection rates.
• Low-density capabilities trigger more generation, oversupplied capabilities are down-weighted, and task length is treated as a separate sampling dimension.

Human annotation

• Human-annotated trajectories from real workflows complement automated rollouts and capture long-tail interaction patterns.
• Validation includes deterministic structural checks and multimodal review of task completion and state transitions.
• Observation repair fixes target leakage from cursor or hover states and incomplete rendering by selecting earlier or later buffered frames.
• A teacher model relabels human recordings with reasoning and natural-language actions while keeping human action parameters fixed.

Verifiable RL tasks

• Each RL task bundle includes an instruction, environment, fixed initial state, reference completion state, and executable verifier.
• The invariant requires the initial state to score 0 and the reference completion to score 1.
• Generation uses capability-guided sampling and decoupled construction of environments and rewards.
• Evaluator refinement uses hard-negative probes, alternative-positive probes, and rollout feedback.
• Tasks are promoted to the RL corpus only after passing invariant checks and both positive and negative evidence tests.

The provided excerpts do not specify exact dataset sizes, training split percentages, or final mixture ratios. Allocation is described through capability-based rebalancing rather than fixed proportions.

OSWorkerBench Benchmark

Composition

• OSWorkerBench contains 100 realistic office tasks across 41 normalized applications and 10 job families.
• All 100 tasks support instruction-only evaluation.
• The benchmark includes two independently annotated, potentially overlapping subsets:
  • 67 Long-Memory tasks requiring delayed reuse of dynamic information or sustained workflow-state tracking.
  • 49 Multi-App tasks requiring transfer of dynamic, multi-field information across at least three logical applications.

Demonstration-guided settings

• Self-demo: 33 tasks are paired with successful strong-agent rollouts of the same tasks.
• Variant-demo: 45 tasks are paired with human demonstrations from semantically related but non-identical source tasks.
• The 33 and 45 are separate demonstration collections, not a partition of the 100 tasks.
• The 45 variant-demo tasks are especially long-horizon, with Kimi-2.6 averaging more than 100 decision turns per task under instruction-only evaluation.

Construction and processing

• Workflows are synthesized from occupations, applications, verified UI capabilities, and target difficulty.
• Difficulty is assigned during synthesis along breadth, depth, and reasoning axes.
• Dense evaluators comprise 1 to 13 checkpoints, with mean 4.86 and median 5.
• Eighty-eight tasks use state-based evaluators over application backends, while 12 use task-specific evaluators for spreadsheets, images, compound documents, or conditional workflows.
• Human reviewers verify instruction clarity, setup completeness, feasibility, checkpoint coverage, and expected partial scores.
• For DemoCUA demonstrations, a capable GUI agent first produces trajectories, which are converted into structured action sequences with annotated screenshots. Human annotators then complete unfinished parts, remove redundant interactions, and retain key actions without revealing task-specific answers.

Coverage statistics

• 99 of 100 tasks require at least two applications.
• The mean is 3.26 applications per task, with a maximum of 7.
• Common hubs include Slack in 65 tasks, Gmail in 34, Google Sheets in 31, Salesforce in 23, and Google Calendar in 19.
• Kimi-2.6 trajectories have a median of 68 observation-to-decision turns and a mean of 88.3; 38 of 100 trajectories reach at least 100 turns.

OSWorkerBench is used for evaluation rather than training, with strict success requiring all final-state conditions and partial progress measured by a checkpoint-weighted score.

Method

The authors define a computer-use task as a pair $\mathcal{T} = (x, \mathcal{E})$T=(x,E), where $x$x is a natural-language instruction and $\mathcal{E}$E is the environment. The agent interacts in discrete decision steps, receiving a screenshot observation $o_t$ot​ and producing a response $y_t = (r_t, a_t)$yt​=(rt​,at​) containing intermediate reasoning and actions to execute.

To train the general computer-use policy, the system employs a training stack combining Supervised Fine-Tuning (SFT) with Agentic Reinforcement Learning (RL). The RL pipeline optimizes the policy for successful task completion using verifiable rewards from the environment.

As shown in the figure below:

The RL system operates in two main phases: rollout and update. During the rollout phase, an Adaptive Sampler selects tasks, prioritizing weak domains to ensure broad coverage. The Policy Server interacts with Online Sandboxes (the Environment Server), executing actions across applications like Web, Mail, and Calendar. These interactions generate trajectories that are collected in a Rollout Pool. In the update phase, an Outcome Verifier inspects the final state of these trajectories to assign binary rewards. To improve credit assignment beyond the final outcome, an optional Process Credit Model (PCM) analyzes the trajectory steps, identifying progress, redundancy, and errors. The RL Trainer then performs asynchronous Group Relative Policy Optimization (GRPO) updates on the Policy Model, utilizing decision-turn centering and token-level normalization to handle varying trajectory lengths and response sizes.

To enable the agent to learn from user-specific procedures, the authors introduce DemoCUA, a module for learning from in-context multimodal demonstrations.

Refer to the framework diagram:

The demonstration process consists of an offline capture pipeline and an online execution loop. Offline, a native recorder captures a sequence of events. A Vision-Language Model (VLM) analyzes these steps to extract semantic intent and groups them into coherent subtasks. Online, the agent receives a live screenshot and a workflow hook from the harness. This hook provides a progress checklist, the current subtask goal, and key milestones without pixel coordinates, guiding the agent to perform actions on the desktop environment.

The integration of this demonstration into the model's context is structured to prevent shortcut learning while providing guidance.

As shown in the figure below:

The context window injects the demonstration blocks into the first user turn. It includes the workflow progress (the subtask checklist with done, current, and todo markers), the current subtask (the specific goal and completion criteria), and the current subtask action list (key milestones). This setup ensures the agent focuses on the active subtask. Crucially, the action list contains only key milestones, forcing the model to infer intermediate steps from the live screenshot rather than blindly copying the demonstration.

Finally, for deployment, the authors designed the UI-Mate App with a modular architecture to separate model logic from desktop control.

The architecture is depicted as follows:

The system comprises four layers. The Frontend manages task control, visualization, and demo capture. The Backend entry validates goals and routes requests. The Harness drives the core agent loop, handling model inference, the observe-reason-act cycle, and context memory. The Bridge adapts these actions to the specific operating system, handling screen observation and input execution. This design allows the agent to operate on local desktops or virtual machines without requiring plugins for installed applications.

Experiment

The paper evaluates UI-Mate against general-purpose and specialized GUI agents on OSWorld-Verified, WindowsAgentArena, and the newly introduced OSWorkerBench, which adds dense checkpoint-based evaluators and human-in-the-loop verification for long-horizon cross-application office workflows. Results show that environment-grounded training lets UI-Mate match or surpass larger general-purpose models and prior specialized agents, with particular gains in OS control, workflow coordination, long-memory, and multi-application tasks, while strict end-to-end success remains limited by late-stage omissions. Demonstration-guided evaluation on OSWorld, OSWorkerBench, and GameDev shows that same-task demos improve task completeness and execution efficiency, and additional analyses indicate that historical reasoning aids inference but can reduce exploration during reinforcement learning.

UI-Mate-27B is competitive with strong general-purpose multimodal models on OSWorld-Verified, outperforming several general-purpose and specialized computer-use agents while remaining below the top closed models. The smaller UI-Mate-9B already matches the larger UI-Mate-27B on operating-system interaction, with scaling gains concentrated in application and workflow tasks. The results suggest that training-data coverage and quality can matter as much as model size for computer-use performance. UI-Mate-27B outperforms general-purpose Kimi-K2.6 and Qwen3.7-Plus as well as specialized ScaleCUA-Qwen3.5, EvoCUA-32B, and UI-TARS-1.5. UI-Mate-27B trails the strongest closed general-purpose models, Claude Sonnet 5 and Claude Opus 4.8, and is slightly below GPT-5.5. UI-Mate-9B surpasses the larger EvoCUA-32B, showing that scale alone does not determine computer-use capability. The 9B and 27B UI-Mate models share the same operating-system score, with the larger model's improvement concentrated in Office, Daily, Professional, and Workflow tasks.

Closed-weight frontier models lead OSWorkerBench under the instruction-only protocol, with GPT-5.6-Sol posting the highest progress and binary success among reported systems. UI-Mate improves substantially over its Qwen base models at both 9B and 27B scales, and UI-Mate-27B slightly surpasses Kimi-K2.6 on overall success and progress while remaining behind frontier models. Its relative strengths appear on Multi-App and Long-Memory subsets, where it outperforms Kimi-K2.6. Closed-weight frontier models hold the top progress scores and binary success rates, with GPT-5.6-Sol leading all reported systems. UI-Mate-27B improves strongly over its Qwen3.6-27B base model and slightly surpasses Kimi-K2.6 on overall success and progress. UI-Mate-27B outperforms Kimi-K2.6 on Multi-App and Long-Memory subsets, indicating stronger cross-application and state-tracking performance. UI-Mate-9B also improves substantially over its Qwen base model and outperforms the specialized ScaleCUA agent built on the same base model.

UI-Mate establishes the strongest open-weight performance on WindowsAgentArena, with UI-Mate-27B leading open baselines and approaching top closed-source systems. Gains persist across model scales, as UI-Mate-9B substantially improves over its base model and a same-scale specialized agent while also surpassing a larger specialized model. UI-Mate-27B outperforms all listed open-weight baselines, including models up to 1T parameters. UI-Mate-9B improves sharply over its Qwen3.5-9B base model and a same-scale specialized agent, and surpasses the larger EvoCUA-32B agent.

Demonstration guidance improves the average GameDev success rate of UI-Mate-27B from 76.76% to 81.15%, a gain of 4.39 percentage points. The largest improvements occur on long-horizon tasks requiring structured fine-grained operations, while tasks already solved perfectly remain unchanged. Demonstrations also shorten average trajectory length, implying better execution efficiency without reducing completion. Demonstrations raise average success by 4.39 percentage points and reduce average trajectory length from 303.6 to 253.1 steps. The largest gains are on long-horizon structured tasks such as godot-04, godot-07, and qgis-01; already perfect tasks show no change, while a few tasks see small declines.

Self-demonstration guidance improves UI-Mate-27B performance on both OSWorld and OSWorkerBench subsets. The gains are larger on OSWorld, where instruction-only performance is lower, and more tasks reach perfect scores under guidance. A small number of OSWorld tasks still degrade, indicating that demonstration guidance can occasionally be misapplied. On OSWorld, self-demo guidance raises average progress by 25.48 percentage points, with improvements on 18 of 30 tasks and four previously unsolved tasks completed perfectly. On OSWorkerBench, self-demo guidance raises average progress by 13.29 percentage points, improves binary success by 18.18 percentage points, and increases perfect-score tasks from one to five.

Across OSWorld-Verified, OSWorkerBench, and WindowsAgentArena, UI-Mate is evaluated under instruction-only protocols and compared against general-purpose and specialized computer-use agents. UI-Mate-27B leads open-weight baselines and approaches top closed frontier models, while UI-Mate-9B matches the larger model on operating-system interaction and outperforms larger specialized agents, indicating that training-data coverage and quality can matter as much as scale. Demonstration and self-demonstration guidance experiments on GameDev, OSWorld, and OSWorkerBench show improved success rates and shorter trajectories, especially on long-horizon structured tasks, with only occasional small degradations.