One-sentence Summary

Researchers from HKUST, CityUHK, FDU, et al. present a systematic survey of memory mechanisms in autoregressive video generation, proposing a unified framework that organizes the literature across Forms, Functions, Operations, Learning, and Evaluation and arguing that effective memory must remain accurate, accessible, and causally influential for long-horizon and interactive generation.

Key Contributions

• The survey presents a systematic review of memory mechanisms in autoregressive video generation and introduces a unified operational definition of memory as persistent historical information maintained across outer autoregressive steps, capable of influencing future generation after the originating evidence is no longer locally accessible.
• It organizes prior work through five complementary perspectives: memory forms, functions, operations, learning, and evaluation, and identifies distinct bottlenecks that can produce similar visual symptoms but require different remedies.
• It synthesizes open challenges including intervention-linked state maintenance, controlled revision and branching, and verifiable evaluation, establishing a structured foundation for memory-conditioned video generation systems.

Introduction

Scalable autoregressive video generation is moving from short clips toward open-ended and interactive world modeling, where models must extend visual sequences while preserving identity, layout, motion, events, and causal state over long horizons. The core challenge is temporal persistence under bounded access to history: once early visual evidence leaves the active context, generation can drift or forget entities, appearance, spatial structure, dynamics, semantics, and action-induced state changes. Prior approaches address this with rolling windows, KV caches, retrieval banks, compressed states, or 3D priors, but the field lacks a paradigm-specific taxonomy and consistent terminology for memory, making systematic comparison difficult. The authors present a unified framework for memory in autoregressive video generation, organizing mechanisms by representational forms, functional responsibilities, operational lifecycles, learning strategies, and evaluation protocols.

Method

The authors introduce a unified memory-conditioned autoregressive framework that augments the bounded local context with a persistent memory state to carry managed historical information across generation steps. At each step, the model forms a query from the current context, retrieves relevant historical information, and integrates it with the local context to generate the next visual unit. Following generation, a write candidate is extracted, the persistent state is updated, and a management operator enforces the memory budget through retention, compression, or revision.

To organize the diverse approaches to implementing this persistent state, the authors propose a carrier-centric taxonomy. Memory carriers serve as the representational objects in which historical information is retained and made available to subsequent generation.

As illustrated above, the framework distinguishes four major types of memory carriers. Visual Memory preserves past frames, clips, or observation-aligned visual representations in pixel or VAE latent space. Implicit State Memory retains history in model-native latent states, such as attention-cache states, recurrent or state-space states, and encoded history states, without predefined semantic structure. Explicit State Memory represents history through structured variables describing entities, scene layouts, dynamics, or topological relations. Adaptive Parametric Memory encodes sequence-specific information in designated parameters or adaptable weights that are updated from prior evidence and influence subsequent generation through the model's forward computation.

Beyond the representational form, the authors define memory functions as the specific generative responsibilities that a persistent state must fulfill after the original evidence leaves the bounded local context. These functional responsibilities correspond to the failure modes of bounded-context rollouts.

The framework identifies five core preservation functions. Identity preservation maintains instance-defining properties to mitigate entity forgetting and appearance drift. Spatial preservation maintains scene structure and viewpoint consistency across navigation and revisitation. Dynamic preservation sustains motion and evolving physical states when earlier evidence is no longer in the active context. Semantic preservation maintains rollout-specific roles, relations, events, and high-level commitments. Causal preservation maintains the delayed consequences of state-changing interventions. A single carrier may support multiple functions, and a single function may be realized jointly by several carriers.

The figure above demonstrates how these memory functions manifest in long-horizon video generation, showing how memory supports identity, spatial, dynamic, semantic, and causal preservation across separated observations to maintain persistent entities, track evolving states, and preserve intervention consequences.

To ensure reliable long-horizon generation, the authors detail how memory behavior is acquired through data, supervision, and closed-loop interaction. Because model-generated frames are written back into memory, prediction errors can compound over extended horizons, making memory learning inherently closed-loop.

The learning process is organized along three dimensions. Memory Learning Objectives define the supervision targets, including output-level prediction, direct memory state supervision using explicit world-state variables or learned representations, and memory dynamics supervision that constrains state evolution over multiple transitions. Memory State Distribution addresses the mismatch between training and deployment states, utilizing teacher forcing, history augmentation to broaden the distribution, and self-rollout to construct states from the model's own autoregressive process. Finally, Memory-Aware Learning explicitly incorporates deployment-time constraints, training the generator to learn how to retain relevant history, compress long histories into bounded states, retrieve information from external stores, and adapt parameter states under specific resource budgets.

Experiment

The evaluation sections argue that visual fidelity and coherence are insufficient evidence of memory, so memory-revealing tests must create an information gap in which a later query requires prior rollout history. They classify benchmarks by evidential strength, identifying entity reappearance, scene revisitation, and out-of-view state evolution as the main direct protocols, while visible-consistency, long-context, and sequence stress benchmarks provide only complementary evidence. Target-specific metrics for identity, geometry, process state, goal continuation, and action effects are meaningful only under such protocols and require paired-history controls, prerequisite checks, component interventions, and evaluator reliability to rule out execution failures and non-memory explanations. The concluding discussion calls for comparable diagnostic profiles that separate retention, state correctness, accessibility, and influence, and that cover underexplored cases such as contradiction, selective forgetting, rollback, and delayed action effects.

The notation summary defines core autoregressive visual memory concepts: full history before generation, model-generated visual units, flattened discrete tokens, continuous latent representations, and bounded local context. The evaluation discussion emphasizes that current memory assessments are strongest for identity recurrence and spatial revisitation, while change-sensitive behaviors remain underexplored. It calls for diagnostic, stage-wise evaluation separating retention, state correctness, accessibility, and generation influence, alongside controlled protocols for memory updates and resource use. Coverage is strongest for identity recurrence and spatial revisitation, whereas correction, contradiction resolution, selective forgetting, stale-state invalidation, rollback, and delayed intervention effects remain underexplored. Future evaluation should move from isolated endpoint scores to diagnostic memory capability profiles that distinguish whether information was retained, kept correct, accessible when needed, and actually used during generation.

The listed visual memory mechanisms retain historical visual evidence in itemized units, with medium interpretability, direct editability, and model coupling. Their storage growth is either bounded or linear, while access keys range from explicit pose or content queries to learned queries or no separately addressable historical unit. Evaluation coverage is strongest for identity recurrence and spatial revisitation, whereas correction, contradiction resolution, stale-state invalidation, selective forgetting, rollback, and delayed interventions remain underexplored. All listed visual memory mechanisms use itemized unit structure and medium interpretability, editability, and model coupling, while storage growth ranges from bounded to linear. Evaluation is stronger for identity recurrence and spatial revisitation than for correction, contradiction resolution, stale-state invalidation, selective forgetting, rollback, and delayed intervention.

These implicit state mechanisms retain history in model-native latent states, including long-context Mamba states, recurrent SemanticPack states, linear-attention states, compressed contextual states, fixed-budget hierarchical history tokens, and gated evolving memory-query states. Most methods use fixed storage and distributed updates, and the listed entries show low interpretability and editability with high memory capacity. The surrounding analysis indicates that evaluation is stronger for identity and spatial revisitation and weaker for contradiction, correction, stale-state invalidation, and delayed intervention. Implicit state carriers include recurrent SemanticPack states, clean-pass-updated linear-attention states, compressed contextual memory states, and fixed-budget hierarchical history tokens. Every listed method uses distributed updates, and almost all use fixed storage; interpretability and editability are consistently low while memory capacity is high. Evaluation coverage is described as strongest for identity recurrence and spatial revisitation, while correction, contradiction resolution, stale-state invalidation, selective forgetting, rollback, and delayed intervention remain underexplored.

Representative identity preservation methods span multi-shot, interactive, and long-video settings, and several combine identity with spatial coverage. Coverage is strongest for identity recurrence and spatial revisitation, while dynamic, semantic, and causal preservation remain less consistently addressed. Evaluation is still limited by endpoint comparisons that do not localize whether memory failed at retention, state correctness, retrieval, or utilization. Identity preservation mechanisms include reference-conditioned multi-shot anchors, content anchors, first-chunk appearance conditioning, retrieved historical VAE blocks, shot cache guidance, and anchor memory with tri-region RoPE. Identity and spatial preservation receive the strongest coverage, while correction, contradiction resolution, stale-state invalidation, selective forgetting, rollback, and delayed intervention effects remain underexplored.

The table classifies representative dynamic-preservation methods for AR video generation by their functional coverage, carrier type, and evaluation setting. The listed methods commonly combine identity and spatial preservation with dynamic preservation, while semantic preservation appears in fewer cases and causal preservation is not marked. The accompanying discussion indicates that identity recurrence and spatial revisitation are the most evaluated capabilities, whereas delayed-action and causal persistence protocols remain less mature. Dynamic-preservation methods frequently also support identity and spatial preservation, but semantic support is less consistent and causal preservation is absent from the listed rows. Latent video and internal state are common memory carriers, with long-context video and interactive settings among the evaluation conditions. The supporting text notes that identity recurrence and spatial revisitation are best covered, while contradiction resolution, stale-state invalidation, and delayed intervention effects remain underexplored.

The surveyed experiments compare itemized memory units, implicit latent-state carriers, identity preservation methods, and dynamic preservation methods across storage, interpretability, editability, and coupling properties. The evaluated settings mostly validate identity recurrence and spatial revisitation, while dynamic, semantic, and causal preservation receive less consistent coverage. Current endpoint-based assessments do not isolate whether failures occur at retention, state correctness, accessibility, or generation influence. Correction, contradiction resolution, stale-state invalidation, selective forgetting, rollback, and delayed intervention remain underexplored.