One-sentence Summary

Researchers at Hong Kong University of Science and Technology propose HOMIE, a human-object centric video personalization framework that unifies inter- and intra-subject settings by integrating a multimodal large language model to extract reference-level relationships, applying global multimodal guidance within self-attention to align MLLM-derived semantic features with VAE tokens, and using modality-reference embeddings to differentiate MLLM and VAE token types and associate intra-subject reference image tokens, achieving state-of-the-art performance across various HOCVP tasks.

Key Contributions

• The paper introduces HOMIE, a unified framework that handles both inter-subject and intra-subject human-object centric video personalization by integrating MLLM knowledge without compromising text encoder controllability.
• A global multimodal guidance mechanism injects MLLM-derived semantic features into video tokens via self-attention, improving alignment with VAE tokens and enabling more accurate interaction modeling between humans and diverse objects.
• Modality-reference embedding differentiates tokens by their input modality and captures intra-subject associations among reference images, strengthening identity fidelity in intra-subject settings. Experiments show state-of-the-art performance across various HOCVP tasks, including logo personalization and multi-view video generation.

Introduction

Human-object centric video personalization (HOCVP) aims to synthesize videos of multiple subjects interacting from reference images, enabling applications like personalized advertising and digital content creation. Existing methods handle inter‑subject scenarios (distinct entities) with difficulty as the number of subjects grows: they produce copy‑paste artifacts, fail to accurately place abstract references such as logos, and lack the reasoning to infer implicit relationships. In intra‑subject settings (multiple references of the same entity, e.g., multi‑view images), they either miss semantic correspondences or lean on costly 3D supervision, resulting in redundancy and limited fidelity gains. Prior attempts to integrate multimodal large language models (MLLMs) either align MLLM features to a constrained text‑embedding space, suppressing visual reasoning, or replace the text encoder entirely, forcing the MLLM to carry the full text‑to‑video control burden and diluting its focus on reference associations.

To overcome these limitations, the authors propose HOMIE, a unified HOCVP framework that preserves the standard text encoder and lets the MLLM concentrate purely on extracting inter‑ and intra‑subject relationships. HOMIE introduces global multimodal guidance, which enriches video tokens with MLLM‑derived temporal interaction context during self‑attention, and modality‑reference embeddings that distinguish tokens by modality while jointly capturing intra‑subject associations. This design improves prompt adherence, subject consistency, and identity fidelity across diverse scenarios, including logo personalization and multi‑view video generation.

Dataset

The authors construct their training data from two open-source subject-driven video datasets, OpenS2V-5M and PhantomData, through a multi-step filtering and curation pipeline. Here’s how the final dataset is built and used:

• Source and initial filtering Videos from OpenS2V-5M and PhantomData are first cleaned by removing low-quality clips based on generic aesthetic scores. Then, using the original annotations (category labels and subject area proportions), clips that focus on backgrounds or very small objects are filtered out, keeping only videos relevant to human-object composition.

• Subset composition The cleaned data is split into two video-consistency subsets:

  • Single-subject: 300,000 samples containing only a human or an object.
  • Multi-subject: 80,000 samples containing both a human and an object.

• Self-curated high-quality addition The authors further curate a separate set of 20,000 high-quality video clips that feature humans interacting with objects. These clips come with precise segmentation masks for both the human and the object, and they are integrated into the multi-subject subset, expanding it to 100,000 samples. From this expanded multi-subject pool, roughly 40,000 high-resolution (720P) clips are selected.

• Special processing for intra-subject references In the self-curated portion, the same object is cropped consistently across different frames. This enables the construction of intra-subject reference pairs that are used during training to improve the model’s ability to maintain subject appearance (referred to as MRE training).

• How the data is used The single-subject and multi-subject subsets serve as the main training data for the human-object composition video generation model. The self-curated subset, with its segmentation masks and frame-consistent object crops, provides the necessary supervision for learning fine-grained subject identity and composition, particularly through the intra-subject reference mechanism. The final mixture of subsets is not specified in terms of training batch ratios, but the 300K single-subject, 100K multi-subject (with 40K high-res), and the intra-subject reference data all contribute to the overall training pipeline.

Method

The authors develop HOMIE based on a DiT-based text-to-video model as the backbone architecture. The framework integrates a 3D Variational Autoencoder (3D VAE) to compress raw video into a compact latent space. A text encoder processes textual prompts, which are injected into the DiT backbone via cross-attention mechanisms to establish text-video semantic alignment. The model is trained using the flow matching paradigm, learning a continuous velocity field to transform samples from a simple prior distribution to the target data distribution along deterministic trajectories. The training objective is defined as:

where $z_{0}$z0​ denotes a sample drawn from a prior distribution, $z_{1}$z1​ is the latent of the target video sample, and $c$c denotes the condition set. The DiT model $v_{\theta}$vθ​ takes the noisy latent $z_{t} = (1 - t)z_{0} + tz_{1}$zt​=(1−t)z0​+tz1​ and learns to predict its velocity at any point $t$t. To enable high-fidelity and controllable human-object complex video generation, the conditioning set is extended as $c = \{c_{\mathrm{txt}}, c_{\mathrm{img}}\}$c={ctxt​,cimg​} to incorporate multimodal data inputs.

As shown in the figure below:

HOMIE accepts a textual prompt $\mathcal{P}$P and a set of reference images $\mathcal{I} = \{I_{n}^{m}\}$I={Inm​} as inputs, where superscript $m$m denotes subject identity and subscript $n$n indexes the image instance. The input images are mapped into a latent space via the pre-trained 3D VAE. To capture complex semantic alignment between visual subjects and textual instructions, a Multimodal Large Language Model (MLLM) $\Phi_{M}$ΦM​ processes both the images and the prompt to extract a multimodal feature $F_{M}$FM​. A lightweight alignment network $\phi$ϕ aligns this conditioning feature with the 3D-VAE token space:

where $\mathcal{P}_{sys}$Psys​ denotes the system prompt. The multimodal feature $F_{M}$FM​ is concatenated with the latent tokens of the target video $F_{v}$Fv​ and reference images $F_{r}$Fr​ to form the unified input $F = [F_{v} \oplus F_{r} \oplus F_{M}]$F=[Fv​⊕Fr​⊕FM​], allowing intrinsic interaction of tokens from different modalities.

To address the distributional gap between the highly semantic MLLM features and the 3D VAE features, the authors introduce Global Multimodal Guidance (GMG). This mechanism explicitly injects semantic information from $F_{M}$FM​ into $F_{v}$Fv​ within the self-attention layers. After projecting the query and key matrices, they are partitioned into components corresponding to video tokens, reference tokens, and MLLM feature tokens. Global representations for the MLLM query $Q_{M}$QM​ and key $K_{M}$KM​ are computed by applying a pooling operation along the temporal dimension:

These global representations are fed into two lightweight projection networks, $\gamma$γ and $\beta$β, to derive scaling and shifting modulation factors. After broadcasting these parameters to align with the sequence length $l_{v}$lv​ of the video tokens, a feature-wise affine transformation is applied to $Q_{v}$Qv​ and $K_{v}$Kv​:

The partitioned query and key components are then concatenated back to execute standard attention computation, effectively infusing high-level semantic multimodal guidance into the video tokens.

To distinguish and align representations across different modalities and capture underlying associations between inter- and intra-subject references, the authors propose Modality-Reference Embedding (MRE). This learnable embedding module consists of modality embeddings (ME) and reference embeddings (RE). Prior to projecting tokens into the stacked DiT blocks, learnable modality embeddings are selectively integrated into each token according to its source modality:

where $* \in [v, r, M]$∗∈[v,r,M] denotes different modality types, and $Proj$Proj maps each modality type to distinct slice indices. Subsequently, reference embeddings are assigned to tokens derived from reference inputs based on their original reference entities:

where $m$m denotes the $m$m-th unique reference entity. References with the same annotation $m$m are assigned the same reference embedding to share identity information, while inter-subject references receive distinct embeddings to denote disparate identities. This facilitates robust disentanglement across diverse modalities and enforces identity-level consistency.

Experiment

HOMIE is evaluated on 200 human-object combinations with multiple reference images against diverse baselines, including both closed-source and open-source models, to assess video quality, text-following, and subject consistency. It achieves state-of-the-art performance in video quality and text alignment among open-source methods, excelling at complex multi-subject generation and logo personalization that require relational reasoning. Ablation studies demonstrate that the GMG module is critical for integrating MLLM knowledge to enable such reasoning, while the MRE module prevents unintended treatment of OCR maps or multi-view references as independent entities. A user study further confirms that HOMIE is consistently preferred across overall quality, text following, and subject consistency.

Among the compared baselines, Kling 1.6 leads in aesthetic and motion quality, while VACE and SkyReels-V3 show strength in text following and object similarity. HOMIE surpasses all listed methods in video quality and text adherence, achieving a 21.8% relative improvement in OCR accuracy over SkyReels-V3. Kling 1.6 attains the highest AES and MS scores, indicating top perceptual quality and smoothness among the baseline methods. VACE records the best DINO-I and OCR accuracy, and SkyReels-V3 achieves the leading object similarity, highlighting their respective text-following and subject consistency capabilities. HOMIE’s OCR accuracy is 21.8% higher than SkyReels-V3, validating effective use of evaluation-dataset OCR maps for subject consistency. The baselines exhibit trade-offs across video quality and text-following dimensions, while HOMIE unifies strong performance across all metrics.

On the intra-subject multi-view HOCVP benchmark, HOMIE (Ours) sets new state-of-the-art scores on both DINO reconstruction and accuracy metrics. The Wan2.2 variant achieves the highest reconstruction score, while Wan2.1 leads in accuracy, each substantially outperforming prior methods such as SkyReels-V3 and VACE. These quantitative gains reflect the model’s ability to faithfully synthesize diverse object views with strong subject consistency. HOMIE’s Wan2.2 variant reaches the best DINO_rec (0.696), a 6.4% relative improvement over SkyReels-V3, demonstrating markedly better multi-view reconstruction. The Wan2.1 variant attains the highest DINO_acc (0.582) among all models, indicating superior subject identity preservation in the generated views.

The full model outperforms all ablated variants, with marked improvements in face similarity and OCR accuracy over a plain baseline that omits multimodal inputs. Replacing the GMG module with direct integration of MLLM features into UmT5 embeddings lowers the overall quality score and fails to enable reasoning for logo personalization, while removing the Modality-Reference Embedding causes a sharp drop in OCR accuracy, confirming its role in linking reference modalities. Without the Modality-Reference Embedding, OCR accuracy falls to 0.376 and reference objects appear unintentionally, showing that MRE captures semantic links among reference images. Directly feeding MLLM features into UmT5 embeddings yields the lowest GMEScore (0.675) and fails to connect logos to semantically relevant objects, highlighting GMG’s importance for reasoning.

Evaluation against strong baselines shows that HOMIE unifies top-tier video quality, text adherence, and subject consistency, avoiding the trade-off observed in prior methods. Multi-view benchmark results further demonstrate HOMIE’s ability to reconstruct diverse object views with faithful identity preservation, setting new state-of-the-art scores. Ablations confirm that both the global–local geometry reasoning module and the modality–reference embedding are essential for linking reference inputs semantically and achieving accurate personalization.