One-sentence Summary

Meituan LongCat Team introduces dNaViT, a Discrete Native-resolution Vision Transformer that combines a frozen pretrained encoder with lightweight residual modulation and grouped product quantization to produce compact discrete visual tokens for unified encoding and high-fidelity decoding, matching or exceeding baselines across 12 visual understanding benchmarks, 4 generation benchmarks, and an image editing benchmark.

Key Contributions

• dNaViT is introduced as a native-resolution discrete visual tokenizer built on a frozen pretrained encoder, applying lightweight residual modulation with a dual-branch objective to preserve semantic structure and fine-grained pixel information within a shared vocabulary.
• The method uses grouped product quantization to scale discrete capacity without inflating spatial sequence length, and progressive training balances semantic and pixel-level reconstruction; a code-conditioned refinement step further improves perceptual fidelity without altering the unified representation.
• Evaluations show dNaViT matches or exceeds task-specific baselines across 12 visual understanding benchmarks, 4 generation benchmarks, and an image editing benchmark while using a substantially smaller training data budget.

Introduction

Language models have scaled successfully by treating text as discrete tokens, which serve as a unified substrate for representation learning and autoregressive modeling. In contrast, modern multimodal systems often use modality-specific continuous representations and heterogeneous objectives, making it difficult to reuse language-model techniques and infrastructure for vision. The key challenge is that discretizing dense visual signals into a finite vocabulary creates a severe information bottleneck, and prior approaches have struggled to preserve both high-level semantics and fine-grained pixel detail in a single discrete code. The authors introduce dNaViT, a discrete native-resolution Vision Transformer that builds on a frozen pretrained encoder, adds a Residual Modulator for pixel-level detail, and applies Grouped Product Quantization with progressive dual-branch optimization. This design yields a shared discrete visual vocabulary that supports reconstruction, classification, multimodal understanding, generation, and editing under a smaller data budget than existing models.

Dataset

The authors construct a tokenization and de-tokenization training corpus from diverse image sources, grouped into three complementary visual-content categories:

• General-domain images: LAION, COYO, DataComp, Recap-DataComp-1B, CapsFusion, and an in-house image corpus. These provide broad coverage of objects, scenes, and visual styles.
• Text-rich images and documents: selected TextAtlas5M subsets, including book covers, long-word images, academic papers, and presentation slides; Chinese and English book/news pages; additional OCR-oriented collections; infographic collections; and ChartGalaxy. These cover dense text, small characters, complex document layouts, charts, and tables.
• Fine-grained and high-aesthetic images: face-focused LAION subsets, SA-1B-derived images, and an in-house high-aesthetic collection. These add detailed facial structures, local textures, and varied image compositions.

No per-subset sample counts or explicit filtering rules are reported in this section. The data is used for tokenizer training with dual-branch supervision, using images exclusively. Processing follows a resolution curriculum: training begins at a fixed 256 × 256 resolution on a limited set of sources for rapid convergence, then transitions to native-resolution training up to 2K on a broader data mixture. During the transition, sampling maintains a balanced distribution across resolution ranges. Images are resized to approximately preserve aspect ratio, align spatial dimensions with the encoder stride, and keep sequence lengths within the token budget. Across all stages, the model is trained on several hundred billion tokens.

Method

The authors introduce dNaViT, a visual tokenizer designed to learn shared discrete codes as a unified interface for diverse downstream tasks. The architecture replaces separate task-specific feature representations by reconciling appearance invariance for understanding with sensitivity to detail for reconstruction. To achieve this, the framework leverages a frozen native-resolution vision encoder and introduces a Residual Modulator (ResMod) to aggregate complementary cues. ResMod fuses intermediate features from multiple encoder depths and learns a spatially varying correction to the final representation. By initializing the residual update to zero, the pretrained features are preserved at the start of training, allowing the module to gradually reweight the semantic base and inject reconstruction-relevant detail.

To retain the exposed semantic distinctions and spatial detail within a shared discrete bottleneck, the enriched features are processed through a grouped product quantizer. Instead of using a prohibitively large monolithic vocabulary, the authors factorize the features along the channel dimension into $G$G disjoint groups. Each group is assigned its own learned codebook, and quantization selects the nearest entry within each group. The resulting discrete index tuples are concatenated to form the final codebook embeddings. This approach expands the quantization capacity combinatorially to $K^G$KG combinations without explicitly storing a massive vocabulary, while keeping the spatial token grid unchanged to support variable-length grids at different image resolutions.

The shared discrete codes are supervised through two specialized decoders. The semantic branch aligns the codes with continuous visual features using a combination of cosine alignment and mean squared error loss, ensuring the retention of the pretrained semantic structure. The pixel branch decodes the embeddings into pixels through token unmerging and spatial expansion, optimized with a composite loss including L1, MSE, LPIPS, perceptual, and Gram matrix objectives. Together, these branches constrain the shared vocabulary to preserve both high-level semantics and fine-grained visual fidelity.

To further enhance perceptual quality, the authors introduce a code-conditioned Refiner. Unlike conventional super-resolution models, this module directly conditions a flow decoder on the discrete visual codes, avoiding information loss from intermediate pixel decoding. The Refiner utilizes a context adapter for global scene-level context and a time-conditioned spatial adapter to inject aligned residuals into the latent input. A multi-step teacher model is trained via flow matching and subsequently distilled into an efficient one-step student using adversarial supervision.

The optimization strategy separates the learning of complementary visual detail from the adaptation of the enriched representation to a discrete bottleneck. Training proceeds through three distinct stages to mitigate interference between objectives. In the first stage, continuous pixel pretraining, the ResMod and pixel decoder are trained to recover the original visual signal while the vision encoder remains frozen. This initializes the residual pathway to expose reconstruction-relevant information.

In the second stage, discrete semantic reconstruction, both the vision encoder and ResMod are frozen. The quantizer and semantic decoder are trained to reconstruct the detail-enriched representation, while the pixel decoder is trained on detached codes to establish an initial mapping from discrete codes to pixels without disrupting semantic learning. Finally, in the joint reconstruction stage, the encoding ResMod is unfrozen and trained alongside a frozen teacher copy to prevent target drift. The semantic branch reconstructs the fixed target from the teacher, while the pixel branch allows gradients to flow back to update the quantizer and the student ResMod, jointly refining the discrete representations for both semantic and pixel reconstruction.

Experiment

The evaluation spans reconstruction, frozen-feature classification, multimodal understanding, text-to-image generation, and instruction-guided editing, with additional analyses of ResMod, discrete versus continuous representations, and pixel decoding. Reconstruction and classification results show that the shared discrete codes preserve global fidelity, fine-grained text and face detail, and linearly accessible semantic information, although small text remains challenging and transfer performance varies by distribution. Multimodal and generative experiments indicate that discrete visual tokens can support competitive reasoning, OCR, document and chart understanding, compositional generation, and editing. Analysis further suggests ResMod retains semantic quality while adding pixel detail, and the flow-based refiner recovers high-frequency information more effectively than direct pixel decoding, with qualitative examples illustrating fine-grained recognition and reasoning.

dNaViT reconstruction quality improves with resolution under a fixed evaluation protocol, with the 64-group variant recording higher PSNR and SSIM at 1024x1024 than the 16-group variant. This suggests the shared discrete representation retains reconstruction-relevant information across resolutions, although training differences prevent isolating the effect of quantization group count. Continuous baselines are reported under source-specific protocols and are not directly comparable. dNaViT-G64 shows consistent PSNR and SSIM gains as resolution increases from 256x256 to 512x512 to 1024x1024. At 1024x1024, dNaViT-G64 scores higher than dNaViT-G16 on both reconstruction metrics, but training configuration and duration differ between the variants.

On TokBench, dNaViT-G64 achieves fine-grained reconstruction results above DC-AE-f32 and comparable to LlamaGen-f8 across text accuracy, text similarity, and face similarity at both 512 by 512 and 1024 by 1024. All evaluated methods improve when resolution increases from 512 to 1024. Stronger continuous models such as SD-3.5 and FLUX.1-dev still lead, leaving a performance gap for small text. dNaViT-G64 scores higher than DC-AE-f32 on all three fine-grained metrics at both evaluated resolutions. The continuous baselines shown improve consistently from 512 by 512 to 1024 by 1024 on text accuracy, text similarity, and face similarity. FLUX.1-dev attains the highest fine-grained reconstruction scores among the listed continuous baselines, followed by SD-3.5.

dNaViT-G64 consistently improves over dNaViT-G16 across small, medium, and large TokBench instance buckets at both tested resolutions. Gains are largest for small and medium instances, while large instances improve more modestly. Small instances remain the most challenging category, with lower text recognition, edit similarity, and face similarity than larger instances even for the stronger variant. dNaViT-G64 outperforms dNaViT-G16 across all instance sizes and metrics at both resolutions. Relative improvements are most pronounced for small and medium buckets, while large buckets show smaller gains. Small instances remain the weakest category, with the largest remaining performance gap in text-related metrics.

Among the listed continuous vision-language models, InternVL3.5-8B achieves the strongest available results on MMMU, MathVista, MathVision, and MMStar, with an especially large advantage on MathVision. RealWorldQA scores are closely clustered across most evaluated models, while LLaVA-OV-2-8B and MiMo-VL-7B-2508 have incomplete benchmark coverage. The accompanying analysis notes that model-level comparisons vary in training data, prompting, and scoring, and suggests discrete visual representations can match continuous features with sufficient training data. InternVL3.5-8B leads the listed continuous models on most STEM and general understanding benchmarks, with the largest gap on MathVision. Discrete visual representations can match continuous features in multimodal understanding as training data scale increases, according to the accompanying analysis.

Frozen visual representations are evaluated by linear probing on ImageNet-1K and by direct transfer on five distribution-shift benchmarks. Continuous dNaViT closely matches QwenViT across all reported benchmarks, while discrete dNaViT is only slightly below its continuous counterpart on ImageNet-1K and leads the evaluated discrete representations. Continuous dNaViT performs on par with QwenViT on ImageNet-1K and all five distribution-shift benchmarks. Discrete dNaViT trails its continuous form by under one percentage point on ImageNet-1K and is the strongest among the evaluated discrete representations.

The experiments evaluate dNaViT's discrete and continuous representations on reconstruction quality, fine-grained TokBench metrics, multimodal understanding, and frozen-feature transfer. Reconstruction improves with resolution, and the 64-group discrete variant consistently outperforms the 16-group variant, especially on small and medium text instances, while remaining behind strong continuous baselines such as FLUX.1-dev and SD-3.5 for fine-grained text. In multimodal and transfer benchmarks, discrete representations approach continuous features as training data scale increases, with continuous dNaViT matching QwenViT and discrete dNaViT leading evaluated discrete models.