One-sentence Summary

Researchers at WeChat Vision, Tencent Inc. propose WeLM-Image, a unified image foundation model for text-to-image generation, instruction-guided editing, and multi-reference synthesis that uses a Mixture-of-Transformers backbone with Mixture-of-Experts modules (31.7B stored parameters and 5.4B activated generation parameters) for single-GPU deployment; in reported overall comparisons, the 1536-resolution model ranks first among the evaluated open-source models on PRISM-Bench, Qwen-Image-Bench, and UnicBench, while the distilled WeLM-Image-Turbo generates a 1024 × 1024 image in 4.98 s on a single NVIDIA H20 GPU.

Key Contributions

• Introduces WeLM-Image, a unified image foundation model designed for single-GPU deployment that processes text, reference images, and target latents as one interleaved sequence for text-to-image generation, instruction-guided editing, and multi-reference synthesis. Its Mixture-of-Transformers backbone combines shared attention with independent understanding and generation parameters, and MoE modules provide 31.7B stored parameters with about 5.4B activated generation parameters per target latent token.
• Describes a generation-oriented data and training pipeline combining a knowledge-grounded visual vocabulary with large-scale visual clustering, bilingual annotations, verified editing supervision, training-time evaluation, progressive training to native 1.5K resolution, and task-specific reinforcement learning followed by noise-aware on-policy distillation.
• Reports that the 1536-resolution model ranks first among evaluated open-source models on PRISM-Bench, Qwen-Image-Bench, and UnicBench. MeanFlow-based few-step distillation yields WeLM-Image-Turbo, which generates a 1024 × 1024 image in 4.98 s on a single NVIDIA H20 GPU using eight guidance-free steps.

Introduction

The authors address image generation and editing for practical, consumer-facing services such as WeChat, where models must handle long bilingual prompts, accurate text rendering, complex edits, multi-reference composition, and large request volumes under tight single-GPU serving budgets. Prior image foundation models incorporate multimodal understanding in different ways, but a scalable unified model must balance language modeling and continuous image generation without optimization interference while preserving fine-grained visual control. The authors present WeLM-Image, a unified image foundation model that treats text-to-image generation, instruction-guided editing, and multi-reference synthesis as interleaved text and image sequences in one backbone. Its dual-path Mixture-of-Transformers and Mixture-of-Experts design separates understanding and generation parameters while shared attention connects them throughout the model, supported by a dedicated VAE and text-compatible 3D rotary positional encoding.

Dataset

The authors build WeLM-Image's dataset from a raw web-scale image-text pool and reorganize it for controllable generation, editing, and composition. The pipeline is summarized below.

Sources and scale

• Raw pool: about 5 billion images from internal data, Internet data, public datasets, and licensed datasets.
• Curated pool: about 0.6 billion images after quality filtering.
• High-resolution share: 59.74% of retained images are at 1024 resolution or higher.

Schema and filtering

• Each sample is normalized into a unified schema that preserves image identity, source metadata, resolution, aspect ratio, raw text, and storage information.
• Quality filtering includes resolution thresholding, blur and clarity assessment, deduplication, and safety filtering with NSFW removal.
• Aesthetic scoring uses an internal Elo-style pairwise preference model, producing a continuous score for later data mixture construction.
• Filtering operators also retain fine-grained quality, resolution, safety, and content signals for downstream sampling, routing, mixture construction, and diagnosis.

Annotations and coverage structures

• A large VLM produces structured annotations describing image content for retrieval, clustering, knowledge-graph alignment, and task construction.
• OCR and watermark operators identify text-heavy content such as screenshots, posters, advertisements, documents, and UI layouts.
• Semantic tags capture entities, attributes, styles, scenes, and other compact visual semantics.
• Knowledge-grounded vocabulary: built from Cilin and WeChat Knowledge Graph, cleaned with LLM checks and Qwen3 embedding plus reranking, organized hierarchically, and expanded with aliases, descriptive phrases, visual keywords, and search expressions.
• Visual clustering: images are encoded with Qwen3-VL-Embedding and grouped by a balanced hierarchical variant of spherical k-means over hundreds of millions of images.

Task-specific subsets

• Text-to-image: KG vocabulary nodes define semantic targets; concept-to-image retrieval plus image-to-concept reverse verification selects candidates; per-concept quotas and globally unique images balance long-tail coverage; captions use randomized bilingual templates across neutral descriptions, concise user-like expressions, and generation-style prompts; OCR and watermark signals are incorporated for text-heavy images.
• Image editing: built from a structured editing ontology covering object addition, object removal, background replacement, local attribute modification, portrait editing, product editing, restoration, and reasoning-oriented edits; VLM-based cluster-to-task assignment balances categories; pairs come from real-image graphs within clusters and reverse construction that keeps real targets; a VLM quality gate verifies source-target differences and prompt diversification rewrites instructions.
• Multi-reference: samples contain 2 to 4 source or reference images, one instruction, and one target; roles include identity, garment, object, scene, style, or layout; 18 tasks are defined; an entity pool of about 750,000 images supports quota-based assembly; reverse construction keeps real targets and forward construction synthesizes targets from real sources; VLM filtering enforces resolution, deduplication, no source-target copies, and usability; the dataset contains several hundred thousand examples.
• Understanding data: includes text-only data and vision-language instruction data from LLaVA-OneVision-1.5-Instruct, FineVision, Honey-15M, and proprietary captioning data; taxonomy covers General VQA, Caption and Knowledge, Chart, OCR, Document, Grounding, Code, Math, Table, Science, Counting, GUI, Medical, Spatial, and Egocentric tasks; filtering removes repetitive text, low-quality images, extreme aspect ratios, and language mixing. Reverse alignment can also turn text-to-image data into vision-language alignment data.

Training usage and mixtures

• Mixture ratios are scheduled by training stage rather than held fixed.
• Early pretraining retains understanding-oriented data to establish the shared multimodal interface.
• High-resolution continuation and SFT concentrate the budget on text-to-image, editing, and multi-reference generation, as detailed in Table 3.
• Training-time evaluation feeds back into filtering thresholds, retrieval queries, sampling quotas, task assignments, and mixture ratios.

Method

WeLM-Image is a unified model designed for image creation, instruction-guided editing, and multi-reference generation. The authors leverage an interleaved multimodal sequence representation combined with a dual-path Mixture-of-Transformers (MoT) backbone. This architecture, referred to as MoT-MoE, incorporates Mixture-of-Experts (MoE) feed-forward layers in both the understanding (UND) and generation (GEN) paths.

Refer to the framework diagram:

Heterogeneous multimodal inputs are normalized into a common interleaved representation. Text and visual elements are independently encoded and projected into model-width token sequences. Understanding tokens, comprising text and image Vision Transformer (ViT) features, are routed to the UND path. Generation tokens, consisting of clean and noisy image Variational Autoencoder (VAE) latents, take the GEN path. Learned routers select experts within the chosen path, while shared attention couples the two paths, which otherwise maintain independent projections, normalization, and experts.

To handle diverse tasks such as text question answering, visual understanding, text-to-image synthesis, and editing, the model adopts a unified interleaved task representation. The model, packer, and training losses operate on a single ordered multimodal sequence rather than task-specific formats.

As shown in the figure below:

Each sample is serialized as typed text and visual segments that preserve source order:

$$\mathcal{S} = [S_1; S_2; \dots; S_K], \qquad S_k \in \{\text{text segment, image segment}\}.$$S=[S1​;S2​;…;SK​],Sk​∈{text segment, image segment}.

In the generation template, reference images contribute adjacent clean-VAE segments (GEN path) and ViT segments (UND path). The instruction takes the UND path, while the noisy target VAE takes the GEN path and receives flow-matching supervision. Setting $N=0$N=0 yields text-to-image generation, whereas $N \ge 1$N≥1 extends the template to editing and multi-reference generation. In the understanding template, an optional input-image ViT segment and the question precede the response, with next-token cross-entropy applied only to the response.

Each image segment is represented through two complementary visual encoders. A ViT encoder captures high-level content such as objects and spatial relations, while the WeLM-Image-VAE maps the image into compact continuous latents retaining visual details for reconstruction and generation. The VAE is trained with a principal-component compression strategy in three stages: reconstruction pretraining, semantic alignment, and text-oriented decoder fine-tuning. The semantic alignment stage jointly trains a lightweight diffusion transformer to recover high-level semantic features from corrupted latents, aligning them with features from a frozen DINOv2 encoder by matching their normalized self-similarity matrices:

$$\mathcal{L}_{\mathrm{rep}} = \left\| \bar{r}_z \bar{r}_z^\top - \bar{r} \bar{r}^\top \right\|_F^2, \quad \bar{r} = \frac{r}{\|r\|_2}, \quad \bar{r}_z = \frac{r_z}{\|r_z\|_2}.$$Lrep​=​rˉz​rˉz⊤​−rˉrˉ⊤​F2​,rˉ=∥r∥2​r​,rˉz​=∥rz​∥2​rz​​.

The backbone contains 27 decoder layers. MoT separates understanding and generation into independent parameter paths while retaining joint attention for cross-modal interaction. To scale efficiently, each MoT path is equipped with an independent MoE. The first layer uses a path-specific dense Feed-Forward Network (FFN), while layers 1 through 26 replace the dense FFN with a path-specific MoE sublayer.

At each decoder layer, hidden states are gathered by path and mapped to queries, keys, and values through path-specific pre-attention RMSNorm and Multi-head Latent Attention (MLA) projections. A single masked scaled-dot-product attention operation is applied across both paths.

Refer to the attention mask diagram:

The block-structured attention mask isolates packed samples and enforces task-dependent visibility. Text follows causal order, while ViT and clean reference VAE tokens use bidirectional attention within their segments. Noisy-target VAE tokens attend bidirectionally within the target latent grid and read all preceding non-target conditions, but their keys and values remain invisible to other segments to prevent target leakage.

Following joint attention, outputs are regrouped by path and passed through MoE feed-forward sublayers. Each path combines 64 routed experts with two always-active shared experts. For the hidden state $h_i$hi​ of token $i$i at path $p$p, the feed-forward output is:

$$\mathrm{FFN}_p(h_i) = \sum_{s=1}^2 E_{p,s}^{\text{shared}}(h_i) + \sum_{e \in \mathcal{T}_p(i)} \alpha_{p,e}(h_i) E_{p,e}^{\text{routed}}(h_i).$$FFNp​(hi​)=s=1∑2​Ep,sshared​(hi​)+e∈Tp​(i)∑​αp,e​(hi​)Ep,erouted​(hi​).

Routing uses top-6 experts for the UND path and top-20 for the GEN path, with a non-gradient correction bias updated based on load statistics to ensure stable expert utilization.

Text and visual tokens share one attention context but require different positional structures. The authors lift rotary embeddings to a single three-axis coordinate system $(t, h, w)$(t,h,w), where $t$t is the sequence-axis coordinate and $h, w$h,w are spatial coordinates.

As shown in the figure below:

The 64-dimensional rotary subspace forms 32 channel pairs, with 12 pairs assigned to $t$t and 10 pairs each to $h$h and $w$w in an interleaved order. For text tokens, the coordinates are $(p, p, p)$(p,p,p), ensuring exact compatibility with the pretrained 1D RoPE. VAE appearance tokens retain the temporal index fixed while adding a centered spatial offset $(t, h, w) = (p, p + \Delta h, p + \Delta w)$(t,h,w)=(p,p+Δh,p+Δw).

The unified sequence is decoded by separate modality-specific heads. Text states on the UND path are trained with autoregressive next-token prediction. Noisy target states on the GEN path are projected into the VAE latent space to parameterize a flow-matching velocity, trained via conditional flow matching:

$$\mathcal{L}_{\mathrm{FM}} = \mathbb{E} \left[ \left\| \hat{v}_\theta - v^\star \right\|_2^2 \right].$$LFM​=E[∥v^θ​−v⋆∥22​].

The joint objective combines cross-entropy, flow-matching, and router regularization losses. The flow-matching loss is normalized at the image level to prevent larger images from dominating the objective.

The authors build a generation-oriented data engine to reorganize web-scale data around controllable generation tasks.

Refer to the data pipeline diagram:

The pipeline refines noisy web data for quality, retaining a curated image pool. It indexes the pool along a knowledge-grounded vocabulary for named visual concepts and large-scale visual clustering for coherent appearance patterns. Task-specific data for text-to-image, editing, and multi-reference generation is constructed on this organized pool, with training-time evaluation feeding back into filtering and mixture ratios.

WeLM-Image is trained through a pipeline of progressive pretraining, instruction Supervised Fine-Tuning (SFT), task-separated reinforcement learning (RL), and distillation stages. Both paths of the dual-path backbone are initialized from a pretrained language-vision checkpoint.

Pretraining progresses through three stages, increasing synthesis resolution from 512 to 1536 and context length up to 64K tokens, while shifting the data mixture toward generation tasks. SFT retains the 1536 resolution and uses a curated instruction mixture.

Post-training optimizes text-to-image and image-to-image editing as separate RL branches using DiffusionNFT. Four VLM rewards measure image-text alignment, aesthetics, text rendering, and visual naturalness for T2I, while a checklist-based reward evaluates editing. On-policy distillation consolidates the capabilities of the two RL experts into a single student checkpoint. Finally, a few-step distillation framework based on MeanFlow compresses the model into a guidance-free 8/4 step model for efficient inference.

Experiment

The experiments evaluate WeLM-Image as a unified generation and editing model through serving efficiency on NVIDIA H20 GPUs, quantitative benchmarks for text-to-image quality, text rendering, and image editing, and qualitative case studies. Efficiency results show that generation and editing latency scale near-linearly with input and output tokens, while the distilled Turbo variant further reduces cost; benchmark results consistently place WeLM-Image among the top open-source models, with particular strength in text rendering, instruction following, and single-image and multi-image editing. Qualitative examples confirm that the model generates stylistically diverse, text-rich images and can perform coordinated multi-instruction and reference-based edits while preserving intended content.

Among the evaluated VAEs, FLUX.2-VAE achieves the strongest reconstruction fidelity on standard benchmarks and the best generation quality on ImageNet. FLUX-VAE is typically the second strongest for reconstruction, while Qwen-VAE is the second strongest for generation. Hunyuan-3.0-VAE shows weaker reconstruction fidelity on standard benchmarks and was not evaluated for generation. FLUX.2-VAE leads reconstruction quality on COCO and ImageNet and also achieves the best FID and IS on ImageNet generation. FLUX-VAE and Qwen-VAE are competitive runners-up in reconstruction and generation, respectively. On in-house poster images, FLUX-VAE and FLUX.2-VAE attain the highest SSIM scores, while FLUX-VAE and Qwen-VAE lead in PSNR. Hunyuan-3.0-VAE has comparatively weak reconstruction results on standard benchmarks and lacks generation evaluation.

The backbone separates understanding and generation into independent parameter paths that share joint attention, with independent mixture-of-experts feed-forward layers in most decoder blocks. The generation path routes through far more experts per token and activates roughly 2.5 times more decoder parameters than the understanding path. The full model includes the ViT encoder and frozen parameters in stored totals but excludes the VAE. The understanding path uses top-6 expert routing per token, while the generation path uses top-20 routing, creating a large per-token activation gap. Activated decoder parameters are about 2.2B for understanding tokens and about 5.4B for generation tokens. Layer 0 uses a path-specific dense feed-forward network, while the remaining 26 layers replace the dense feed-forward network with a path-specific MoE sublayer. Each path has its own MoE experts rather than a shared expert pool, preserving task separation across understanding and generation.

The training recipe spans progressive pretraining at increasing resolutions and context lengths, followed by supervised fine-tuning. Early stages keep a modest share of understanding data to establish cross-modal alignment, while later high-resolution and SFT stages restrict the mixture to text-to-image, editing, and multi-reference generation. Optimizer settings remain largely fixed across stages, with a lower learning rate used for SFT. Resolution and context length grow progressively from the 512 stage through the 1024 and 1536 stages. Understanding supervision is retained in the 512 and 1024 stages but removed from the 1536 and SFT mixtures. The SFT stage lowers the learning rate while optimizer, weight decay, and gradient clipping settings stay consistent with earlier stages. Parallelism keeps the product of FSDP shard and context-parallel size fixed at 32, with expert parallelism set to 1.

Latency measurements on NVIDIA H20 GPUs show that few-step guidance-free configurations provide much lower end-to-end latency than standard classifier-free guidance models. For standard 50-step generation, latency scales close to linearly with output pixels, while editing latency grows steeply with the number of input images in several baselines. SenseNova-U1-8B-MoT is the fastest listed 50-step model and shows the flattest multi-image editing curve, and WeLM-Image is reported to be faster than Qwen-Image-2512 at a matched step budget. The 4-step Lightning configurations cut end-to-end latency by roughly an order of magnitude compared with their standard 40 to 50 step counterparts for text-to-image and editing. Editing latency for Qwen-Image-Edit rises steeply as input images increase from one to four, while SenseNova-U1-8B-MoT stays nearly flat over the same range.

WeLM-Image demonstrates top-tier open-source performance across text-to-image generation and text rendering benchmarks, with particular strength in compositional generation and vision-language evaluated quality. Closed-source models are treated separately from open-source rank highlighting. The results position WeLM-Image as a leading open-source model on both mainstream and emerging evaluations. WeLM-Image ranks among the top-tier open-source models across mainstream and emerging generation benchmarks. It achieves the best open-source overall score on PRISM-Bench and leads in Imagination, Style, and Composition. On OneIG-Bench, it records the second-best alignment score and third overall among open-source models. The WeLM-Image family performs strongly on GenEval, with the Turbo-8step variant reaching a perfect single-object generation score.

The evaluated experiments cover VAE quality, model architecture, training configuration, inference latency, and generation benchmarks. FLUX.2-VAE is the strongest VAE overall, with FLUX-VAE and Qwen-VAE as competitive alternatives, while the backbone separates understanding and generation into independent expert paths with generation activating substantially more parameters. Training progresses through increasing resolutions and context lengths while removing understanding data in later stages, few-step and Lightning configurations greatly reduce latency, and WeLM-Image ranks among top open-source models with leading compositional and vision-language quality.