One-sentence Summary

Researchers from Chinese Academy of Sciences, Microsoft Research, and other institutions propose BiVidGen, a hybrid MLLM-DiT video-generation framework in which an MLLM autoregressively generates discrete semantic visual tokens using an EMA-based tokenizer and a DiT renders video conditioned on both text and these tokens through multi-layer cross-attention, improving semantic alignment and temporal coherence on VBench-Long.

Key Contributions

• A systematic study of MLLM and DiT integration for video generation identifies discrete semantic visual tokens from an EMA-based tokenizer, causal autoregressive generation, and explicit visual-token conditioning as effective design choices.
• BiVidGen is introduced as a hybrid framework in which an MLLM generates semantic visual tokens and a DiT renders videos conditioned on both text and those tokens through multi-layer cross-attention.
• Experiments on VBench-Long show that BiVidGen improves semantic alignment and temporal coherence over a fine-tuned DiT baseline, supporting explicit MLLM visual planning as a stronger intermediate interface than text-only conditioning.

Introduction

Video generation has shifted toward diffusion-based transformers as the dominant high-fidelity synthesis backbone, while hybrid image models show that coupling multimodal large language models with diffusion decoders can improve text understanding and visual alignment. However, in video generation such hybrid designs remain less explored: existing unified video models typically freeze the MLLM and rely on lightweight adapters or learnable query tokens, treating it mainly as a feature encoder rather than exploiting its generative ability for structured semantic planning. The authors systematically examine three design levels: what intermediate representation should pass from the MLLM to the DiT, how the MLLM should generate it, and how the DiT should consume it. Their proposed BiVidGen uses discrete semantic tokens from an EMA-based tokenizer, causal autoregressive generation in the MLLM, and explicit multi-layer cross-attention conditioning in the DiT, yielding improved temporal consistency, causal transitions, and semantic alignment on VBench-Long.

Method

The authors formulate a hybrid MLLM-DiT architecture for video generation. Given a text prompt $T$T, instead of directly modeling the generation process as $p(X|T)$p(X∣T), they introduce an intermediate representation $D$D produced by the MLLM prior to diffusion rendering. Specifically, the MLLM generates $D \sim p_\phi(D|T)$D∼pϕ​(D∣T) from the input prompt, and the DiT models $X \sim p_\theta(X|D, T)$X∼pθ​(X∣D,T) conditioned on both the MLLM-produced representation and the original text prompt.

The authors focus on explicit visual tokens as the intermediate representation $D$D. Since the original MLLM is not designed to directly generate visual-semantic tokens, they adapt its output space by quantizing continuous semantic visual features extracted by the MLLM visual encoder into discrete visual tokens.

As shown in the figure below:

The authors compare five quantization variants. Let $C$C denote the continuous visual feature sequence extracted by the MLLM visual encoder and $\{ \mathbf{e}_n \}_{n=1}^N${en​}n=1N​ denote a learned visual codebook. Each feature vector $\mathbf{C}_k$Ck​ is quantized to its nearest codebook entry as $\mathbf{D}_k = \mathbf{e}_{d_k}$Dk​=edk​​, where $d_k = \arg\min_n \| \mathbf{C}_k - \mathbf{e}_n \|_2$dk​=argminn​∥Ck​−en​∥2​. For embedding-based tokenizers, they optimize the standard vector-quantization objective:

where $\mathrm{sg}[\cdot]$sg[⋅] denotes the stop-gradient operator and $\beta$β controls the commitment loss. They additionally consider auxiliary supervision variants, including an MLLM understanding loss $\mathcal{L}_{\mathrm{MLLM}} = - \sum_{j=1}^M \log p_\psi(y_j | y_{<j}, \mathbf{D})$LMLLM​=−∑j=1M​logpψ​(yj​∣y<j​,D) where a frozen MLLM predicts the paired text sequence from quantized features, and a reconstruction loss $\mathcal{L}_{\mathrm{rec}} = \| g_\omega(\mathbf{D}) - X \|_2$Lrec​=∥gω​(D)−X∥2​. Alternatively, they employ an Exponential Moving Average (EMA) update scheme without gradient-based optimization. This maintains the exponential moving averages $\mathbf{s}_n$sn​ of accumulated features and assignment counts $c_n$cn​:

where $\mu$μ is the decay factor. The codebook entry is updated as $\mathbf{e}_n = \frac{\mathbf{s}_n^{(i)}}{c_n^{(i)} + \delta}$en​=cn(i)​+δsn(i)​​. Finally, they explore hierarchical token structures where features from different layers of the visual encoder are quantized.

To enable explicit visual-token generation, the authors fine-tune the MLLM to acquire this capability, investigating different attention mechanisms and token formats.

As illustrated in the figure below:

They design four representative architectures. The first maintains causal attention in the MLLM, treating discrete text tokens and visual tokens as a unified autoregressive sequence. The second applies bidirectional attention to discrete visual tokens, where masked positions are randomly selected but kept consistent across frames. The third adopts bidirectional attention over continuous visual tokens corrupted with noise, training the model to predict added noise under a diffusion objective. The fourth maintains causal attention for multimodal token processing and introduces a flow matching head that uses the MLLM visual outputs as conditions to generate final continuous visual tokens.

The authors analyze how intermediate tokens are exposed to the DiT renderer, categorizing strategies into natural-language prompt refinement, implicit latent bridging, and explicit visual-token conditioning.

Refer to the diagram below:

For explicit visual-token conditioning, they investigate token format, injection mechanism, and injection depth. They compare discrete and continuous visual tokens. They study self-attention, where generated visual tokens act as structural blueprints and directly participate in denoising dynamics, and cross-attention, where tokens serve as conditioning signals that guide the denoising process externally. They also explore multi-layer injection to introduce visual guidance into multiple DiT blocks for more effective utilization.

After systematically analyzing these design choices, the authors present the final architecture, BiVidGen.

As shown in the framework diagram:

BiVidGen consists of an EMA-based tokenizer, an MLLM-based semantic generator, and a DiT-based renderer. For MLLM training, they fine-tune the model on paired text-image or text-video data. A resolution-frame prefix is prepended to the visual token sequence to improve control. The visual tokens from the EMA-based tokenizer are appended after text tokens and enclosed by and markers. They optimize using standard next-token prediction with teacher forcing, applying cross-entropy loss only to the visual tokens and the token.

For DiT training, the model is conditioned on discrete visual tokens $D$D. $D$D is projected to the DiT hidden dimension and injected into each DiT block via cross-attention:

where $b$b indexes the DiT blocks. By introducing visual guidance at every block, the DiT integrates semantic conditions throughout the denoising process. To preserve the pretrained diffusion prior, the DiT backbone is frozen, and only the newly added cross-attention modules and projection layer are trained with the standard diffusion MSE loss. During inference, the MLLM autoregressively generates visual tokens starting from the marker, conditioned on the user prompt. The resulting sequence is fed into the diffusion generator to produce the final visual content, retaining the original 3D RoPE positional encoding for stable spatio-temporal modeling.

Experiment

These experiments compare intermediate representations between an MLLM and a video diffusion model, using Qwen3-VL-2B and Wan2.2 with VBench-Long for generation and DAVIS for reconstruction. The results show that prompt refinement helps, latent bridging underperforms due to feature mismatch, and explicit discrete visual tokens injected through multi-layer cross-attention provide the strongest semantic and generation quality gains. Ablations further indicate that the EMA-based codebook best balances reconstruction and code utilization, causal autoregressive modeling of discrete tokens outperforms bidirectional attention, and self- or cross-attention fusion are similarly effective. Qualitative analysis shows improved causal transitions and fine-grained prompt following, while efficiency remains acceptable with a modest latency increase that can be reduced by lowering denoising steps.

Prompt refinement improves over the DiT-only baseline, mainly through stronger semantic fidelity and text alignment. Latent bridging underperforms the baseline on both quality and semantic measures, consistent with a mismatch between MLLM hidden states and the expected DiT conditioning. Explicit discrete visual tokens provide the largest gains, especially when fused through multi-cross-attention, which yields the highest semantic and total scores. Prompt refinement raises semantic and total scores over the text-conditioned DiT-only baseline. Discrete visual tokens with multi-cross-attention achieve the best overall performance, while latent bridging scores fall below the baseline.

The comparison evaluates visual token codebooks by entropy, active ratio, top-token concentration, and quantization error. The EMA-based codebook achieves the best balance, with the lowest quantization error and highest entropy, indicating accurate reconstruction and diverse code usage. In contrast, the embedding-only codebook shows limited utilization and higher error, while adding an LLM improves code activity but substantially increases quantization error. The EMA-based codebook records the lowest quantization error and the highest entropy among the evaluated codebooks. The embedding-only codebook has very low active ratio and high quantization error, indicating poor code utilization and reconstruction accuracy. The embedding plus LLM codebook attains the highest active ratio but also the highest quantization error.

Discrete visual token inputs consistently outperform continuous inputs on the VBench-Long benchmark under the same training budget. Causal attention for vision tokens provides stronger semantic following than bidirectional attention, with only a negligible difference in visual quality. These results support modeling discrete visual tokens autoregressively to produce high-quality intermediate representations for diffusion. Discrete visual tokens achieve higher semantic and total scores than continuous tokens across attention types. Causal attention improves semantic following over bidirectional attention while visual quality remains nearly unchanged. Discrete tokens with causal attention deliver strong semantic performance and overall quality, supporting autoregressive visual token modeling.

EMA-based tokenizers provide stronger reconstruction on DAVIS than embedding-based alternatives, with cross-attention injection yielding the best PSNR, SSIM, and LPIPS. Adding a decoder reconstruction loss improves reconstruction relative to the plain embedding baseline, whereas combining an embedding codebook with LLM features performs worse. Self-attention and cross-attention are both viable fusion mechanisms. EMA with cross-attention obtains the highest PSNR and SSIM and the lowest LPIPS among all variants. For the same self-attention injection, EMA improves over embedding-based codebooks across all reconstruction metrics. Adding a decoder loss improves reconstruction over the plain embedding baseline; embedding plus LLM has the weakest reconstruction. Cross-attention yields a slight reconstruction gain over self-attention for the EMA codebook.

Across the evaluated configurations, EMA-based tokenization achieves the strongest semantic scores and overall generation scores, while embedding-based variants tend to have slightly higher visual quality but weaker semantic alignment. Self-attention and cross-attention injection with EMA tokens perform comparably, and adding decoder or LLM modules to embedding tokens lowers semantic and total scores. EMA tokenization yields the highest semantic and total scores, with self-attention and cross-attention performing comparably. Embedding-based tokenization obtains slightly higher visual quality but lower semantic scores than EMA. For embedding-based tokens, adding a decoder or LLM injection module reduces semantic and total performance, with the LLM variant showing the weakest overall results.

The experiments evaluate how different visual conditioning strategies, codebook designs, attention patterns, and fusion mechanisms affect semantic alignment, reconstruction quality, and generation performance. Discrete visual tokens, particularly with EMA-based codebooks and multi-cross-attention or causal attention, consistently provide the strongest semantic following and overall generation quality. Prompt refinement improves over the text-only baseline, while latent bridging and embedding-plus-LLM variants underperform, and embedding-based codebooks tend to trade slightly better visual quality for weaker semantic alignment.