One-sentence Summary

H3-WORLD, an efficient framework from Tencent, the National University of Singapore, and The Hong Kong Polytechnic University, turns the MiniMax-H3 video generator into an interactive world model by aligning structured language instructions with temporal video latents and employing temporal attention routing to prevent control leakage, achieving precise character and camera control with only 0.199% trainable parameters.

Key Contributions

• The paper demonstrates that large video generators like MiniMax-H3 already exhibit coarse language-based control, providing a foundation for interactive world modeling.
• H3-WORLD is introduced to convert this language understanding into precise world control by representing character and camera actions as textual instructions and applying temporal attention routing, without needing dedicated action-specific modules.
• With only 8,000 gameplay samples and 0.199% trainable parameters, H3-WORLD achieves effective control, preserves generation quality, and generalizes to unseen actions and visual scenarios.

Introduction

Pretrained video generators like diffusion transformers have become powerful foundations for simulating visually rich worlds, but they lack precise action interfaces for interactive control. Prior work typically adds dedicated control modules, such as action embeddings or camera geometry, that require extra supervision and risk disrupting pretrained capabilities. The authors observe that large models like MiniMax-H3 already respond to coarse textual motion commands, and they leverage this to build H3-WORLD. The framework expresses character and camera actions as compositional text instructions aligned with video latent intervals, using temporal attention routing to maintain control precision. This lightweight adaptation (only 0.199% parameters) enables effective interactive world modeling while preserving generation quality and generalizing to unseen actions and scenes.

Method

The authors propose H3-WORLD, a framework designed to generate future video latents $\mathbf{V}_{1:K}$V1:K​ conditioned on an initial observation $I_0$I0​, a static semantic condition $s$s, and a scheduled action sequence $\mathbf{a}_{1:K}$a1:K​. The system builds upon MiniMax-H3, a pretrained bidirectional audio-video foundation model that jointly denoises the complete future horizon. The pretrained parameters $\theta$θ of the backbone remain frozen, while lightweight adaptation parameters $\phi$ϕ are learned specifically for action control. The conditional generation process is formulated as:

MiniMax-H3 processes text, image, audio, and video tokens within a shared sequence. H3-WORLD retains the native audio stream and focuses its architectural modifications on visual action control.

As shown in the figure below, the overall architecture comprises three primary components. The semantic action interface converts character and camera controls into compositional textual instructions. Latent-aligned temporal binding associates each instruction with a specific video latent interval. Finally, single-egress routing maintains this association throughout the H3 backbone, while Low-Rank Adaptation (LoRA) learns the corresponding action-conditioned visual dynamics.

The semantic action interface translates external discrete control states into the native text-conditioning space of the model. Each control state contains recorded character and camera keys alongside a binary camera-speed flag. These keys are aggregated within the interval of each native H3 video latent to form a latent-level state. To expose the compositional structure of the control space, the authors separate character control from camera control. For the $k$k-th video latent interval, the action is defined as $\mathbf{a}_k = (u_k, c_k)$ak​=(uk​,ck​), where $u_k \in \mathcal{U}$uk​∈U represents character-control commands and $c_k \in \mathcal{C}$ck​∈C represents camera-control commands. Each control pair is mapped to a short textual instruction via clause concatenation:

This representation preserves the factorization of the original action space. However, the training data covers only a sparse and highly imbalanced subset of the valid combinations.

The figure below summarizes the joint distribution of character and camera clauses in the training set, highlighting that the top 20 combinations account for 71.4% of all action prompts, while 52 structurally valid combinations remain entirely unseen during training.

To establish a precise correspondence between each instruction and the generated video, the authors introduce latent-aligned temporal binding. Each action prompt $p_k$pk​ is encoded by the shared H3 encoder $\mathcal{E}$E and processed by a shared two-layer token refiner $\mathcal{R}$R to produce action tokens $\mathbf{A}_k = \mathcal{R}(\mathcal{E}(p_k))$Ak​=R(E(pk​)). The token refiner employs block-diagonal attention, allowing tokens within the same action span to communicate bidirectionally while processing different action spans as separate sequences.

The initial observation $I_0$I0​ and static semantic condition $s$s are jointly processed to produce static semantic tokens $\mathbf{S}$S, while a visual VAE maps $I_0$I0​ to a first-frame condition $\mathbf{C}_0$C0​. These visual conditions and generation targets are packed into a single sequence:

where $\mathbf{P}$P denotes masked padding tokens. Mirrored temporal positions are assigned to the action spans such that $\tau(\mathbf{A}_k) = \tau(\mathbf{V}_k) - \Delta$τ(Ak​)=τ(Vk​)−Δ for $\Delta > 0$Δ>0. This construction preserves the text-before-video ordering of the pretrained model and provides a consistent temporal alignment cue.

To restrict information flow and prevent action spans from communicating directly with unmatched video latents during bidirectional self-attention, the authors implement single-egress routing. A deterministic routing mask ensures that when an action span $\mathbf{A}_k$Ak​ serves as an attention key, it can only be read by tokens within the same action span and its matched video latent $\mathbf{V}_k$Vk​. When $\mathbf{A}_k$Ak​ serves as a query, it retains access to the static context, first-frame condition, native audio context, its own tokens, and the matched video latent, while access to other spans is masked. All video latent spans retain the original bidirectional attention pattern, allowing visual effects to propagate through video-to-video attention for motion continuity.

LoRA is utilized to learn the action-conditioned transformations along these permitted routes. Low-rank updates are applied to the QKV and output projections in the H3 self-attention blocks, as well as to the two-layer token refiner. The H3 backbone, encoder, visual VAE, and all remaining components remain frozen, and the training process retains the native H3 denoising objective without introducing additional learnable parameters for the routing mask or span partition.

Experiment

The evaluation uses gameplay clips from ABot-World-Explorer-500h, training LoRA adapters on H3 and comparing against frozen pretrained and direct action-conditioning variants. Experiments show that LoRA adaptation enables temporally precise control over camera and character motion, while pretrained H3 only follows coarse global instructions and per-latent interfaces without adaptation remain unresponsive. The text-based action interface outperforms direct feature conditioning, and the adapted model generalizes compositionally to unseen action combinations and visually to out-of-distribution scenes while preserving scene layout and style.