One-sentence Summary

National University of Singapore researchers propose a parallelized autoregressive framework for dense video captioning that exploits weak local dependencies across temporally distinct events to restructure the causal dependency graph, enabling lossless parallel decoding via a latent global planning mechanism that learns event-level structure and produces compact tokens encoding global inter-event causality while adaptively aggregating event-level audio-visual semantics, and an event-factorized parallel decoding mechanism that balances local focus with global inter-event awareness, thereby improving efficiency and performance in omni-modal event grounding and captioning.

Key Contributions

• PadCaptioner restructures the causal dependency graph of dense video captioning by exploiting weak inter-event dependencies, enabling lossless parallel decoding and faster inference.
• A latent global planning mechanism learns event-level structure and produces compact tokens encoding global inter-event causality, adaptively aggregating audio-visual semantics to guide the restructuring.
• An event-factorized parallel decoding mechanism balances local event coherence with global inter-event awareness; experiments on omni-modal benchmarks show improved decoding speed, temporally grounded captioning accuracy, and generalization to other video understanding tasks.

Introduction

Dense video captioning requires jointly localizing multiple events in untrimmed videos and generating temporally grounded descriptions, a task crucial for embodied agents, video search, and multi-event generation. Existing autoregressive video large language models perform well but suffer from prohibitive decoding latency as the number of events grows, while diffusion-based parallel decoders fail to close the performance gap with autoregressive methods, particularly in video understanding. The authors propose PadCaptioner, a parallelized autoregressive framework that restructures the token dependency graph by exploiting weak inter-event dependencies: a latent global planning stage first generates compact event tokens to coordinate the video’s temporal structure, and then event-conditioned subchains are decoded in parallel across events while preserving sequential generation within each event. This yields a 3.8× speedup and state-of-the-art grounding and captioning accuracy, with strong generalization to other temporally grounded omni-modal tasks.

Method

The authors first outline the standard Omni-LLM architecture, which consists of modality-specific encoders and an LLM decoder. Given a video with synchronized audio, the visual and audio streams are encoded into token sequences. To preserve temporal alignment, the video is partitioned into segments according to the visual frame sampling interval, where each segment corresponds to one sampled frame and its temporally aligned audio span. Tokens within the same segment are grouped as multimodal chunks, yielding a sequence that is concatenated with the textual instruction embeddings to form the prefix for the LLM decoder. The model then generates output tokens autoregressively: $L_i \sim \mathbb{P}(L_i \mid P, L_{<i})$Li​∼P(Li​∣P,L<i​) where each generated token is sampled from the distribution generated by the LLM decoder, under the condition of prefix input and all previously generated tokens.

The authors argue that the strictly sequential dependency chain adopted in existing frameworks is not uniformly necessary for every token, and that tokens with weak dependencies can be generated in parallel without sacrificing performance. Video content is inherently organized into temporally distinct events, providing a natural structural basis for dependency decomposition. Fine-grained semantic interactions are typically concentrated within event boundaries, while cross-event relations are largely governed by higher-level global semantics. Consequently, enforcing detailed token-level causal dependencies across events introduces redundant serialization. Motivated by this observation, the authors propose a parallelized autoregressive framework that restructures the dependency graph to enable efficient parallel generation while maintaining temporally grounded caption quality.

As shown in the figure below:

The framework comprises two core components: a latent global planning mechanism that discovers a valid event-level structure to guide dependency reformulation and aggregate event-centric audio-visual semantics, and a parallelized decoding strategy that balances local semantic coherence with global structural awareness.

In the latent global planning stage, conditioned on the multimodal prefix, the model autoregressively generates a sequence of global event tokens. Each token is intended to encode a temporally coherent event as a condensed semantic representation in the latent space. The number of events is not predefined but adaptively inferred, enabling the model to capture the intrinsic temporal structure of each video. Technically, the latent global token is implemented by introducing a special token into the vocabulary. To ensure the planned event-level abstraction provides structurally valid and semantically sufficient anchors for subsequent parallel decoding, the authors introduce an explicit event grounding constraint and adaptive semantic aggregation. The grounding constraint aligns the representation of each generated global event token with multimodal prefix features within its corresponding ground-truth temporal segment while suppressing similarity to features outside this interval. Formally, they optimize: $\mathcal{L}_{\mathrm{gnd}}^{G^i} = \frac{1}{T} \sum_{t=1}^{T} \mathrm{BCE}(y_t^i, \mathrm{sim}_t^i)$LgndGi​=T1​∑t=1T​BCE(yti​,simti​) where BCE denotes the binary cross-entropy loss, and the binary indicator specifies whether a time segment lies within the ground-truth event interval. For semantic aggregation, the authors explore three strategies: uniform aggregation via mean pooling, adaptive aggregation through a lightweight learned scoring head, and attention-guided aggregation derived from the attention map between text query tokens and multimodal tokens.

Guided by the globally coordinated event tokens produced during the planning stage, the authors reformulate the original fully serialized token dependency into a structured, event-factorized dependency graph. Instead of modeling caption generation as a single causal chain over all tokens, they decompose it into multiple weakly coupled, event-conditioned subchains so that different subchains can be decoded in parallel. At each decoding step, local tokens across all event branches are decoded simultaneously under a conditional distribution: $\{L_j^i\}_{i=1}^K \sim \mathbb{P}(\{L_j^i\}_{i=1}^K \mid P, G^{1:K}, \{G^i, L_{<j}^i\}_{i=1}^K)${Lji​}i=1K​∼P({Lji​}i=1K​∣P,G1:K,{Gi,L<ji​}i=1K​) where the shared multimodal prefix and the global event tokens encode global inter-event structure. Despite adopting an event-wise parallel decoding paradigm, the approach preserves the autoregressive property within each subchain to maintain strong local semantic coherence.

Refer to the comparison of decoding strategies:

Based on the reformulated dependency graph, the authors derive a modified causal attention scheme that enables structured parallel decoding. During local generation, each token is permitted to attend to the shared multimodal prefix and all global event tokens to ensure global inter-event awareness, as well as the event anchor token and previously generated local tokens within the same branch to preserve strong intra-event autoregression. Meanwhile, tokens from different event subchains are explicitly masked from attending to one another, thereby eliminating unnecessary cross-event dependencies. To accommodate the inherent variability in response lengths across different subchains while preserving synchronized parallel decoding, the authors modify the positional encoding scheme such that tokens at the same decoding step across different subchains share identical position indices. This design allows a subchain to terminate independently upon generating an end-of-sequence token without affecting other active branches. During training, a padding-based alignment strategy is adopted for subchains shorter than the maximum length, which encourages each subchain to terminate promptly.

Experiment

The experiments evaluate PadCaptioner, a fine-tuned 3B omni-modal model, on dense video captioning benchmarks (LongVALE, ChronusAV, YouCook2) and additional temporally grounded tasks to assess both accuracy and generalization. The model consistently surpasses prior state-of-the-art approaches while offering significantly faster decoding, primarily due to its parallel decoding strategy. Ablation studies validate that latent event planning, event-factorized attention, and adaptive semantic aggregation all contribute to better event grounding, richer captions, and improved efficiency, and qualitative results confirm the model's ability to leverage audio-visual cues for coherent event segmentation and description.

PadCaptioner, a 3B model trained on only 12K videos, sets new state‑of‑the‑art on dense video captioning benchmarks LongVALE and ChronusAV. It outperforms larger models including 7B ChronusOmni and 30B Qwen3‑Omni across all metrics, with particularly large gains in event grounding and captioning quality. The results highlight an effective balance between accuracy and parameter efficiency. PadCaptioner (3B) achieves the highest scores on LongVALE and ChronusAV, surpassing all previous methods by clear margins. It improves F1 event grounding by 6.7% and Sim captioning quality by 6.1% over the best prior model on LongVALE. Despite being a 3B model trained on a smaller budget, it outperforms larger 7B and 30B models, demonstrating both accuracy and parameter efficiency.

PadCaptioner, a 3B model, substantially outperforms the 7B ChronusOmni on both dense video captioning benchmarks while achieving around 3.7× faster total decoding and 3× faster per-token decoding. Compared to a lightweight 3B baseline, PadCaptioner delivers higher caption quality and lower per-token latency, though its total video decoding time is longer due to generating more informative captions. On LongVALE and ChronusAV, PadCaptioner improves F1 and Sim scores over the 7B ChronusOmni while reducing per-token decoding latency by a factor of three. When normalized by generated tokens, PadCaptioner is faster than the video-SALMONN-2+ baseline and produces more expressive captions, as reflected by higher evaluation scores.

On the visually-centric dense video captioning benchmark YouCook2, the 3B PadCaptioner model achieves new state-of-the-art zero-shot results across all evaluation metrics, outperforming every prior method including much larger 7B-scale models. Its strong CIDEr and SODA_c scores indicate high-quality, descriptive captions without any task-specific fine-tuning. PadCaptioner (3B) surpasses all previous models, including 7B-scale ones, on F1, CIDEr, and SODA_c under zero-shot inference. It improves CIDEr to 31.7 and SODA_c to 8.2, substantially outperforming the previous best omni-modal model, ChronusOmni (7B), which achieves 27.3 CIDEr and 6.8 SODA_c.

On the Long-VALE benchmark, PadCaptioner outperforms all prior 7B-parameter models on both Omni-TVG temporal grounding and Omni-SC segment captioning, achieving the highest mIoU and best captioning metrics. The consistent gains across these two distinct tasks demonstrate strong generalization to diverse temporally grounded audio-visual scenarios beyond dense video captioning. PadCaptioner (3B) attains a 45.7 mIoU on Omni-TVG, surpassing the next best 7B model ChronusOmni by over 11 points. For Omni-SC, PadCaptioner delivers the highest BLEU-4, ROUGE-L, and CIDEr scores among all compared methods, accompanied by the best segment-level mIoU.

On the ChronusAV benchmark, ARC-HY-Video delivers dramatically better retrieval performance than all other 7B models, while its captioning scores remain competitive with the leading approaches. The results show a clear retrieval advantage that does not carry over to captioning metrics, where other models often produce higher BLEU, ROUGE, METEOR, and CIDEr scores. ARC-HY-Video more than triples the retrieval accuracy ([email protected]) of the next best model on both video-to-text and text-to-video tasks. On captioning metrics, LongVALE-LLM and Qwen2.5-Omni frequently outperform ARC-HY-Video, with LongVALE-LLM leading on V2T CIDEr and T2A ROUGE-L, and Qwen2.5-Omni achieving the highest BLEU-4 on T2V.

PadCaptioner, a 3B model trained on just 12K videos, is evaluated against much larger models on dense video captioning benchmarks (LongVALE, ChronusAV, YouCook2) and on temporally grounded tasks such as Omni-TVG and Omni-SC. It consistently sets new state-of-the-art results, outperforming 7B and 30B competitors in caption quality, event grounding, and decoding speed, thereby demonstrating strong parameter efficiency. In contrast, a separate model (ARC-HY-Video) shows a large retrieval advantage on ChronusAV but does not match the leading captioning scores of other models.