One-sentence Summary

Researchers at Microsoft Research propose a framework that transforms static single-turn tasks into dynamic multi-turn conversations where user intent evolves across turns—incrementally revealed, revised, and redirected—while preserving original evaluation protocols to reuse existing benchmarks as controlled testbeds, revealing that strong static performance does not transfer to this evolving-intent setting, with substantial drops across model families, and highlighting a critical gap in LLMs’ ability to faithfully track and act on evolving user intent, a capability invisible to static evaluation yet critical for future collaborative agents.

Key Contributions

• The paper introduces a framework that converts any verifiable single-turn task into a controlled multi-turn environment by anchoring the final turn to the original task and backward-synthesizing preceding turns with three forms of intent evolution: incremental reveal, revision, and task switching.
• The method preserves the source dataset’s evaluation protocol, allowing the agent’s final action to be verified using the original verifier without any additional annotation or LLM-based judgment.
• Experiments across math, text-to-SQL, search, and coding benchmarks show that frontier and open-source LLMs experience substantial performance drops when user intent becomes dynamic, exposing a fundamental gap invisible to static evaluation.

Introduction

The authors address the shift from single-turn LLM chatbots to agents that collaborate with users over long, evolving conversations, such as in vibe coding or iterative document editing. In these real-world settings, user intent is not static; it is disclosed incrementally, revised, or pivoted across turns. Existing benchmarks remain largely single-turn, where the task is fully specified up front, and even newer multi-turn evaluations resort to LLM-based judging, short user turns, and limited control over user behavior, missing the broader dynamics of real interactions. The authors propose a framework that converts any verifiable single-turn dataset into a multi-turn environment with evolving intent. By anchoring the final user intent to the original single-turn answer, the framework synthesizes preceding turns through three controllable dynamics (underspecification, revision, and task switching) while preserving the dataset’s verifiable ground truth, enabling scalable, long-horizon evaluation without additional labeling or judge models.

Dataset

The authors construct a simulated dataset to capture evolving user intent over long, multi-turn conversations. Here is a breakdown of the dataset’s composition, processing, and usage:

• Source and composition The dataset is built from a single‑turn verifiable dataset. An “anchor intent” is extracted from that source, representing the final, verifiable goal of a user journey. From this anchor, the authors retrospectively generate a sequence of preceding intents, creating a multi‑turn trajectory that simulates how a user’s needs might evolve step by step.

• Verifiability inheritance Because the anchor intent comes from a verifiable source, the entire simulated trajectory inherits this property. Every turn remains automatically checkable, which is crucial for reliable evaluation and training.

• Processing pipeline

  1. Anchor extraction – A single verifiable intent is selected from the source dataset.
  2. Retrospective construction – Preceding intents are crafted backwards from the anchor, forming a coherent history of shifting user goals.
  3. Trajectory simulation – The full sequence is assembled into a multi‑turn dialogue, preserving the original verifiability.

• Usage in the paper The dataset is used to train and evaluate models that must handle long‑horizon interactions with evolving intent. The simulated trajectories provide a controlled, scalable environment where the model’s ability to track and respond to changing user needs can be measured automatically.

Method

The authors formalize user intent as a controllable, structured state with transition dynamics that govern its evolution across conversational turns. This structured formulation enables precise control over the information revealed by the user, moving beyond fixed and under-specified states to capture dynamic behaviors such as information revision and task pivoting. At any turn $t$t, the user's intent is defined as:

where $f_t$ft​ represents the target function the user wants to accomplish, $\mathcal{C}_t$Ct​ is the set of arguments for that function, $\mathcal{C}_t^{\mathrm{rev}} \subseteq \mathcal{C}_t$Ctrev​⊆Ct​ tracks the arguments already revealed to the agent, and $y_t$yt​ is the ground-truth answer.

To model how this intent evolves, the authors propose three distinct transition types.

First, argument reveal occurs when the user discloses a previously unrevealed argument while keeping the underlying function and existing arguments fixed. Second, argument revision happens when the user changes the value of at least one already revealed argument, requiring the agent to update its belief rather than anchor to previous values. Third, function switch involves the user pivoting to a completely different task, where shared arguments carry their values over to the new function.

To simulate these evolving intents over long-horizon interactions in a scalable and automatically verifiable manner, the authors construct a multi-stage simulation framework.

The process begins with Intent Extraction. Since per-turn intent annotations are expensive to collect, the authors leverage existing single-turn verifiable datasets. From each problem-answer pair $(q, y^*)$(q,y∗), they prompt an LLM to extract the source function $f^*$f∗ and its argument set $\mathcal{C}^*$C∗. This triple serves as the anchor intent for the final turn of the simulated conversation, ensuring the agent's final action can be scored against the original dataset verifier.

Next, the framework performs Retrospective Expansion to synthesize the preceding intents that lead to this anchor. For argument revision, the authors generate counterfactual values for the source arguments. For each source argument $c_i^*$ci∗​, an LLM generates a counterfactual value $c_i^{\mathrm{cf}}$cicf​ that the user can later revise back to the source. For function switches, they synthesize a predecessor function $f^{\mathrm{pre}}$fpre and its arguments $\mathcal{C}^{\mathrm{pre}}$Cpre such that $\mathcal{C}^{\mathrm{pre}} \cap \mathcal{C}^* \neq \emptyset$Cpre∩C∗=∅. This procedure can be applied recursively to build longer chains of predecessor functions.

Finally, the authors execute Situated Simulation to generate the multi-turn conversation. A scheduler distributes the transition events across $T$T turns based on a set of consistency rules. These rules ensure the final turn matches the anchor intent with all conditions revealed, and require a function switch to occur only after the current task is fully specified. A renderer then constructs the natural language user response for each turn based solely on the intent update $\Delta \mathcal{I}_t$ΔIt​, rather than the full intent. This renderer concatenates the updated function and arguments with domain-appropriate discourse prefixes to produce realistic dialogue.

Experiment

The evaluation tests LLM agents in multi-turn evolving intent scenarios across four benchmarks, using transitions like reveal, revision, and switch. Strong single-turn performance fails to transfer, with accuracy dropping sharply as transitions accumulate, especially for function switches. Ablations reveal that composing diverse transitions and increasing source task difficulty further degrade performance, while simple memory aids only partially recover it. Overall, agents struggle to maintain accurate intent tracking and integrate context across turns, underscoring the need for better belief state management.

Large language models consistently lose accuracy when user intent evolves across multiple turns, even when they perform strongly on fully specified single-turn tasks. The degradation is most severe in search and software engineering benchmarks, where some models completely fail despite having high single-turn scores. More tool calls do not compensate for the challenge of tracking changing intent, as accumulated context becomes a distractor. Relative accuracy drops from single-turn to evolving-intent settings are substantial across all models, with some models losing over 30 percent of their single-turn performance on BIRD-SQL and over 58 percent on BrowseComp+. On SWE-Bench, GPT 5.1 and Grok 4.20 achieve 0 percent accuracy under evolving intent, timing out or exhausting the tool-call budget, while maintaining 72 and 84 percent single-turn accuracy respectively.

Models consistently lose accuracy when moving from single-turn to multi-turn evolving-intent interactions, with function switches hurting most. Composing multiple transition types generally amplifies the decline, indicating that intent tracking deteriorates as conversational dynamics become more complex. The impact is especially severe on SWE-Bench Verif., where any function switch reduces GPT 5.1 to zero accuracy. Function switch transitions cause the largest drops from single-turn accuracy across all datasets, more so than argument reveal or argument revise alone. Compositions that include a function switch, such as revise+switch and reveal+revise+switch, usually degrade performance further, though BIRD-SQL sees a partial rebound for GPT 5.5. On SWE-Bench Verif., GPT 5.1 fails completely (0% accuracy) under any scenario containing a function switch, while GPT 5.5 retains most of its single-turn performance.

Turn-wise intent tracking on GSM8K conversations with GPT 5.1 shows that argument reveals and changes are tracked nearly perfectly, while function switches cause substantial accuracy degradation. The drop is particularly severe when multiple function switches occur, with accuracy falling from 89% to 82%. Argument reveals and changes are tracked almost perfectly, with accuracy at or above 96%. Function switch tracking accuracy drops sharply from 89% for one occurrence to 82% for two occurrences.

The experiments assess large language models on single-turn and multi-turn benchmarks where user intent evolves via argument reveals, revisions, and function switches. While models track argument updates with high accuracy, function switches cause the most severe performance degradation, and combining multiple transition types amplifies the decline. The impact is especially pronounced in search and software engineering tasks, where some models entirely fail to track changing intent despite strong single-turn results.