One-sentence Summary

Researchers from the University of Oxford and National University of Singapore propose Mental World Modeling (MWM), a generic theoretical framework that makes mental variables like beliefs and intentions core components of world models maintaining a coupled physical-mental state, and instantiate it as MENTIS, a training-free, fully inspectable baseline that decomposes the process into state parsing, target-observation generation, action decomposition, coupled physical and mental transition, and branch-level value evaluation, demonstrating through experiments on a manually constructed multimodal dataset of text, image, and sounding-video stories with eight modern LLM-based world models that explicitly modeling mental state is essential for predicting human decisions and revealing current bottlenecks in mental world modeling.

Key Contributions

• Introduces Mental World Modeling (MWM), a generic framework that makes mental variables explicit core components of a world model by maintaining a coupled physical-mental world state, rendering target-specific partial observations, and simulating joint updates of both components when an action is taken.
• Presents MENTIS, a training-free and fully inspectable baseline that operationalizes MWM through state parsing, target-observation generation, action decomposition, coupled physical and mental transition, and branch-level value evaluation.
• Evaluates on a manually constructed, quality-controlled dataset of situated decision scenarios (text, image, and sounding-video) with 8 modern LLM-based world models, showing that explicit mental modeling is essential for predicting human decisions. Deeper analysis reveals that the main bottleneck is transition simulation against the human reference.

Introduction

World models are crucial for agents that must predict, plan, and intervene in dynamic environments. Current families of world models excel at capturing physical, visual, and spatial dynamics, but they treat human agents merely as moving objects and leave out the mental states—beliefs, intentions, emotions, norms, and relationships—that shape human behavior. Prior cognitive and Theory-of-Mind research often evaluates mental reasoning in isolation, detaching it from the continuous co-evolution of physical and mental scenes. The authors address this gap with Mental World Modeling (MWM), a framework that jointly represents physical and mental dynamics, renders a target agent’s partial observation from the coupled state, and simulates how candidate actions change both the material scene and the mental-social configuration, offering a formal foundation, a training-free benchmark implementation (MENTIS), and empirical evidence that explicit mental world modeling is necessary for predicting human decisions.

Dataset

The authors introduce Menti-Bench, a small, process-complete evaluation testbed designed to measure target-centric world modeling (MWM) in multimodal systems. Unlike outcome-only benchmarks, every record includes gold annotations for intermediate reasoning steps, making it suitable for both standard and oracle evaluation.

• Composition and sources

  • 448 records evenly distributed across three modalities:
    • Text: 320 short narrative stories.
    • Image: 100 records, each containing a sequence of one to five images accompanied by a brief scene caption.
    • Sounding video: 28 clips where dialogue and ambient audio carry part of the decision evidence.
  • Each record presents a situated story, a designated target agent, a minimal decision question, and six candidate action options written in natural language.
  • Scenes span four categories (interpersonal, object/resource, spatial/perceptual, risk/norm) and five everyday domains; 78% involve at least two characters, requiring mental inference about other agents.

• Filtering and quality control

  • Construction prioritizes shortcut control over raw size.
  • For text records: adversarial option balancing ensures near-uniform distribution of gold answers across the six positions; gold and distractor options are length‑matched; blind options‑only probes verify that correct answers cannot be guessed from surface cues alone.
  • For image and video records: each item passed per-record human quality checks. Only 28 of 50 produced sounding‑video clips were retained, discarding examples where audio evidence was insufficient or ambiguous.

• Gold annotation schema

  • Every record carries the full MWM process annotation: the joint current state $s_t^\star$st⋆​, the target observation $o_t^{\epsilon,\star}$otϵ,⋆​, one successor state per option, and the final action. In total, 2,688 successor states are annotated.
  • Annotations follow the state taxonomies defined in the paper and are strictly reserved for evaluation and oracle‑intervention studies; a standard run only sees the public scene, target, question, and options.

• How the data is used

  • Menti-Bench serves exclusively as a validation testbed. There is no training split; models are evaluated in a zero‑shot or oracle‑assisted setup directly on the benchmark.
  • The process‑level gold enables the authors to measure not only final action accuracy but also the quality of intermediate mental‑state predictions, testing whether a system actually constructs the variables that the MWM framework postulates.

Method

The authors propose Mental World Modeling (MWM), a target-centric, action-conditioned world-modeling framework designed to simulate social decision-making. Unlike standard physical world models that focus solely on environmental dynamics, MWM maintains a joint physical-mental world state. The state space factors as $S = S^{\text{phy}} \times S^{\text{ment}}$S=Sphy×Sment, where the physical component stores entities, relations, and environmental conditions, and the mental component stores latent mental-social variables such as beliefs, attention, goals, intentions, emotions, preferences, norms, role relations, and atmosphere.

Refer to the framework diagram.

The MWM process distinguishes two computational roles: the target agent and the world model. The target agent operates from a first-person, partial observation $o_t^\epsilon$otϵ​, while the world model acts as a third-person, generative simulator that maintains the joint state $s_t$st​ and predicts the next state $s_{t+1}$st+1​ conditioned on the target's action $a_t^\epsilon$atϵ​. The physical component of the state captures what exists and how it is configured.

As shown in the figure below:

The mental component captures latent variables that explain how agents interpret the physical scene.

As shown in the figure below:

The dynamic process at each time step consists of three coupled functions. First, the observation generation function $\Omega_\epsilon$Ωϵ​ renders the target's first-person observation from the current joint state, filtering information based on the target's perceptual access and social perspective. Second, the action proposal function $\Pi_\epsilon$Πϵ​ generates a candidate action from this observation. The action is represented as a coupled pair of a physical carrier and a mental-social meaning. Third, the state transition function $T_\theta$Tθ​ predicts the next joint state. This transition is factorized into physical and mental channels, where the physical transition is constrained by the physical carrier and the mental transition depends on both the carrier and the intended meaning.

To operationalize this theoretical framework, the authors introduce MENTIS, a modular, inspectable, and training-free baseline implementation. MENTIS is designed to expose the latent reasoning path rather than directly outputting an answer. The system converts an input scene into a structured current state, renders the target pseudo-agent's observation, parses each option into an action branch, simulates physical and mental successor states in parallel, and evaluates the resulting futures to select the final action.

As shown in the figure below:

The MENTIS inference pipeline follows six stages. In the first stage, the State Parser maps the raw scene into the joint current state $\hat{s}_t = (\hat{s}_t^{\text{phy}}, \hat{s}_t^{\text{ment}})$s^t​=(s^tphy​,s^tment​). In the second stage, the Observation Generator renders the target's first-person observation $\hat{o}_t^\epsilon$o^tϵ​ from $\hat{s}_t$s^t​, ensuring that information inaccessible to the target is filtered out. In the third stage, the Action Decomposition module converts each natural-language option into a structured candidate action with explicit physical and mental components. In the fourth stage, the World State Transition module predicts the successor state for each candidate action. The physical and mental channels are predicted from the shared time-$t$t inputs and merged into the joint successor state $\hat{s}_{t+1}^k$s^t+1k​. In the fifth stage, the Value Evaluator scores each imagined future along three normalized criteria: mental consistency, physical plausibility, and social appropriateness, while also emitting a safety veto flag. Finally, in the sixth stage, a deterministic decision module selects the highest-value branch under fixed tie-breaking rules and maps it back to the option label. Every stage writes its artifact to the run directory, enabling component-level evaluation and ablation studies.

Experiment

The evaluation uses the Menti-Bench process-annotated testbed and a structural ladder of eight language models, ranging from direct answering to full mental world modeling with physical and mental state parsing, observation rendering, and branch simulation. A necessity ladder demonstrates that every added modeling commitment improves target-action prediction, with the full pipeline outperforming direct answering across all models, and channel ablations confirm that both physical and mental channels plus their coupling are required. Oracle-based bottleneck localization identifies transition simulation as the dominant error source, while scenario and modality analyses show that gains are largest for interpersonal decisions and that structured modeling eliminates modality gaps by genuinely leveraging visual, temporal, and auditory evidence. The overall finding is that explicit mental world modeling provides consistent, modality-general improvements, with future work needed on simulating coupled world changes.

Mental world modeling provides the largest gains in scenarios that require reasoning about hidden mental states, such as interpersonal decisions, and it eliminates the performance gap between text, image, and video inputs. The structured pipeline actively uses modality-specific evidence, with visual and auditory channels contributing substantially to the final performance rather than relying on textual priors. The improvement from full MWM over direct answering is greatest for interpersonal decisions and smallest for object/resource scenes, confirming that the benefit is largest where mental variables govern the choice. The initial text-video performance gap disappears under the full MWM system, and channel interventions show that visual and audio evidence are genuinely used, with the visual stream being the most important modality.

MWM organizes social intelligence around two computational roles—a target agent that acts from first-person partial observations and a world model that maintains and simulates a joint state—interacting through three coupled variables. All three variables explicitly split into physical and mental-social parts, making mental-state modeling a native capability. Empirically, this design produces its largest gains on interpersonal decisions, where hidden mental variables control the outcome, and closes the modality gap by converting text, visual, and audio evidence into a shared structured state. World state is a third-person latent variable that combines a physical scene with mental-social variables. Target observation is a first-person rendering with distinct physical and mental channels, making it partial and perspective-dependent. Actions couple a physical carrier with mental or semantic content. Interpersonal decisions show the largest gain from full MWM because they depend most on hidden mental variables.

A structured ladder of mental world modeling reveals that every added commitment improves target-action prediction, with the full pipeline (MENTIS) achieving the highest performance across all models. The options-only baseline is low and nearly constant across model capabilities, confirming that the tasks cannot be solved without story understanding, while direct answering with multiple reasoning chains still falls short of the full system. Oracle interventions show that simulating state transitions is the primary bottleneck, accounting for the largest share of the remaining gap to human performance. Moving from direct answering to explicit reasoning (S1→S2) yields an average gain of +11.3 F1, and the full world-model pipeline (S6) outperforms self-consistency sampling (S3) by 10 points, even when the weakest model under S6 outperforms the strongest under S3. Gold transition simulation alone recovers 45% of the human performance gap, and combining all four oracles reduces the gap to 1.5 points, indicating that future improvements should target the transition model rather than state representation or action decomposition.

The average final-action F1 score rises monotonically from 31.3 for the choice-only floor to 87.9 for the full mental world modeling pipeline, with every added rung improving performance. The largest gains come from reading the story and explicit chain-of-thought reasoning, while structured state modeling and simulation provide further boosts that direct answering cannot match even with more compute. The benefit of world modeling is greatest for interpersonal decisions and is modality-agnostic, closing the gap between text, image, and video inputs. Reading the story alone lifts average F1 by 32.0 points, the single largest improvement across the ladder. Chain-of-thought reasoning adds 11.3 points, the second largest gain, and the subsequent modeling rungs contribute smaller but consistent increments. The weakest model using the full pipeline scores 84.9, surpassing the strongest model using only self-consistency (83.6), showing that structured world simulation provides gains beyond what repeated sampling can achieve. Interpersonal decisions see the largest gain from direct answering to full modeling (+26.4), while object/resource scenes gain the least (+14.0), indicating the benefit is largest when hidden mental variables drive the decision. The modality gap present in direct answering (text 70.8, video 64.8) disappears under the structured state and full pipeline, with all modalities scoring above 90, demonstrating that the modeling pipeline converts diverse inputs into a common format. Channel ablations cause larger drops for the full pipeline than for direct answering, confirming that the world model actively uses visual, auditory, and temporal evidence rather than relying on textual priors.

Oracle interventions reveal that transition simulation and state parsing are the dominant bottlenecks in the mental world modeling pipeline. Providing gold transitions recovers the largest single-stage gain, while combinations of gold intermediate stages recover most of the human performance gap, confirming that errors concentrate in simulating how the world changes rather than in representing its current state. Gold transitions alone close 45% of the gap to human performance, making transition simulation the largest single source of error. Replacing all intermediate artifacts with gold annotations lifts the pipeline to within 1.5 points of the human reference, showing that 81% of the remaining gap originates in the intermediate stages, with sub-additive gains indicating overlapping errors.

The experiments evaluate a mental world modeling pipeline that parses multimodal stories into structured states and simulates transitions to predict target actions, validating its design across text, image, and video inputs. Ablation and ladder studies show that each modeling commitment improves performance, with the largest gains on interpersonal decisions that depend on hidden mental states, and the method eliminates the initial modality gap by actively using visual and auditory evidence. Oracle interventions reveal that simulating state transitions is the primary bottleneck, accounting for 45% of the remaining human performance gap, while combining gold intermediate stages recovers nearly all of it, confirming that future improvements should focus on transition modeling rather than state representation.