One-sentence Summary

Meshy AI proposes Meshy T2, a fast native mesh generation framework that combines a vertex-set mesh VAE encoding meshes into continuous per-vertex latent tokens without quantization with a coarse-to-fine cascade of image-conditioned voxel flow and mesh flow models, achieving state-of-the-art geometric fidelity and end-to-end image-to-mesh generation in a median of 6 seconds.

Key Contributions

• A nearly lossless vertex-set mesh VAE encodes a complete mesh into one continuous latent token per vertex and decodes vertex coordinates, edge connectivity, and face winding order in a single pass, preserving high-precision geometry, artist-authored topology, and part structure without vertex quantization or coincident-vertex welding.
• A two-stage flow-matching pipeline for direct image-to-mesh generation: an image-conditioned voxel flow produces a coarse occupancy scaffold, and a mesh flow populates it with per-vertex latent tokens under optimal-transport-assigned positional encodings, completing end-to-end synthesis in a median of 6 seconds.
• Face-count control is provided by specifying a vertex budget before decoding, which directly governs the expected face count through the triangle-mesh Euler relation and eliminates the need for post-hoc simplification.

Introduction

Polygonal meshes are the backbone of real-time 3D applications, yet producing compact assets that balance geometric fidelity with low vertex budgets and clean, artist-style topology remains a manual, slow process. Existing high-quality 3D generators typically rely on implicit or volumetric representations and extract dense, near-uniform meshes through iso-surfacing; these require post-simplification that distorts sharp features and part boundaries. Direct mesh generation as sequence modeling, while yielding compact topology, forces an unnatural 1D ordering that makes inference costly and risks invalid surfaces from sequential errors. Recent diffusion and flow-based methods address this with parallel generation, but they often quantize vertices or reconstruct faces heuristically, leading to artifacts, and they primarily target retopology of a known surface rather than image-to-mesh generation. The authors introduce Meshy T2, a flow-based framework that preserves nearly lossless vertex-set geometry and connectivity through a bespoke mesh VAE, supports direct image-to-mesh creation in under ten seconds, and gives users explicit control over the face budget with no post-hoc simplification.

Method

The authors propose Meshy T2, a system that generates meshes in a latent space where every token corresponds to exactly one vertex. This representation is established by a vertex-set mesh VAE. Generation itself is a coarse-to-fine cascade of two flow-matching models, instantiated with the linear interpolation schedule of Rectified Flow. Given a reference image, a voxel flow first sketches the overall shape as a $64^3$643 occupancy scaffold; a latent flow then populates this scaffold with per-vertex latent tokens, guided jointly by the image, the voxel scaffold, and the requested vertex budget. Decoding the generated latent set with the VAE decoder yields the final mesh.

The VAE encodes a triangle mesh $M = (\mathcal{V}, \mathcal{F})$M=(V,F) into a latent set $\mathcal{Z} = \{z_i\}_{i=1}^V$Z={zi​}i=1V​ with exactly one token $z_i \in \mathbb{R}^C$zi​∈RC per vertex, and decodes both the vertex positions and the topology from $z$z alone. Unlike prior mesh generators that quantize vertex coordinates, this VAE regresses positions continuously and keeps distinct tokens for coincident vertices, preserving high-precision geometry and artist-authored topology.

The encoder produces one latent token per ground-truth vertex from two inputs: a sparse voxel context summarizing the surface, and one query token per vertex. The context is produced by a local PointNet pooling surface samples into features on a $256^3$2563 sparse voxel grid. Each vertex query is initialized from Fourier features of its continuous position. The queries are refined by cross-attention into the voxel context, followed by interleaved graph-attention and self-attention layers. The decoder is a pure set decoder that consumes latent tokens without positional encoding. It predicts a continuous position $\hat{x}_i$x^i​, an edge embedding $e_i$ei​, and a face embedding $f_i$fi​ for every vertex. Edge prediction uses a spacetime view of adjacency, scoring every vertex pair with a Minkowski-style logit:

Face prediction assembles oriented faces by predicting the cyclic order of the triangle fan for every vertex, represented as a successor mapping $\pi_i$πi​ on the neighbors $N(i)$N(i). To handle open boundaries, the domain of $\pi_i$πi​ is extended with a NULL element $\emptyset$∅. The face head predicts a soft version $P_i$Pi​ of $\pi_i$πi​ using Sinkhorn iterations. At inference, the decoder outputs are converted into an explicit mesh by thresholding the edge logits and rounding the soft successor mappings into hard permutations.

To address the difficulty of generating the per-vertex latent set directly from a single image, the authors split generation into two stages. The first stage synthesizes a coarse geometric scaffold: a binary occupancy grid $\mathbf{O} \in \{0, 1\}^{64 \times 64 \times 64}$O∈{0,1}64×64×64. Generation runs in the continuous latent space of a pretrained Voxel VAE to remove spatial redundancy.

The Voxel VAE is a dense 3D convolutional VAE. The encoder compresses the occupancy grid through two stride-2 stages into a spatially factorized Gaussian posterior at resolution $16^3$163. The mirrored decoder maps a latent grid back to occupancy logits through two 3D pixel-shuffle upsampling stages. Once trained, the VAE is frozen, and every occupancy grid is represented by its posterior mean $\mathbf{z}_{\text{voxel}} = \boldsymbol{\mu}_\phi(\mathbf{O}) \in \mathbb{R}^{8 \times 16^3}$zvoxel​=μϕ​(O)∈R8×163.

The scaffold generator is a Transformer velocity field $f_\theta$fθ​ over the latent grid. The posterior mean is flattened into $16^3 = 4096$163=4096 tokens, each keeping its three-dimensional coordinate. The noisy tokens are processed by a stack of time-modulated Transformer blocks. Self-attention with 3D positional encoding models dependencies among spatial cells, while the reference image, encoded by a frozen DINOv3 backbone, is injected through cross-attention. The model is trained with velocity-prediction flow matching. At inference, the generated latent grid is decoded by the frozen Voxel VAE and thresholded into the binary occupancy scaffold.

The second generative stage operates on the latent sets produced by the Mesh VAE. The authors train a flow-matching model to map Gaussian noise to the clean latent set $\mathbf{x}_0 = \{z_i\}_{i=1}^N$x0​={zi​}i=1N​ under image, voxel, and vertex count conditions.

Image conditioning uses a frozen DINOv3 image encoder, and voxel conditioning starts from the $64^3$643 binary occupancy grid, encoded by the Voxel VAE encoder into a $16^3$163 latent grid. To control the face count indirectly, generation is conditioned on the vertex count. Since exact-count control is difficult to learn, the authors relax it into range control by randomly appending zero-valued pad tokens to the latent set during training. The flow model is a single-stream DiT where latent, image, and voxel tokens are concatenated and processed jointly by self-attention. To handle the unordered nature of latent tokens, the authors assign each latent token a position from a deterministic Sobol point set. During training, real vertices are matched to Sobol candidates by an optimal-transport assignment minimizing squared Euclidean cost. At inference, the Sobol point set itself provides the latent token positions. The model is trained with velocity-prediction flow matching, using classifier-free guidance by dropping conditions during training.

Experiment

The evaluation uses a curated benchmark of 115 diverse assets to compare pipelines on high-poly retopology and image-to-mesh generation under tight face budgets and robustness constraints. Ablations confirm that an optimal-transport position encoding (Sobol OT) dramatically improves geometric fidelity and mesh topology over standard alternatives, driving the largest validation gains. In retopology, Meshy T2 achieves the best geometric alignment and artist-ready topology while running an order of magnitude faster than competing methods and with full reliability. For image-to-mesh generation, it delivers leading semantic alignment and completes in a few seconds with perfect success, surpassing both diffusion and autoregressive baselines in overall quality and practicality for interactive asset creation.

Ablating position-encoding strategies for the vertex-set mesh VAE reveals that solving a full optimal transport assignment between Sobol candidates and vertex coordinates substantially improves all validation metrics. Chamfer distance drops by 45% compared to skipping transport entirely and by 23% relative to a cheaper Morton-order pairing, while Hausdorff distance shrinks more than threefold, confirming that the transport step, not just Sobol sampling, drives the geometric accuracy gains. Full Sobol optimal transport reduces Chamfer distance by 45% versus index-based encoding and by 23% versus Sobol candidates paired in Morton order. Hausdorff distance under Sobol OT is over three times lower than with no transport and 58% lower than with Morton pairing, showing a decisive improvement in worst-case surface alignment.

Meshy T2 achieves the best geometric fidelity and topology in high-poly retopology, delivering the lowest Chamfer and Hausdorff distances and the highest normal consistency, while completing every asset in a median of 3 seconds. MeshAnything V2 and BPT show competitive geometry but are more than an order of magnitude slower. DeepMesh and MeshSilksong suffer from low success rates and weak geometric agreement, making them unreliable for this task. Meshy T2 retopologizes dense 100k-triangle meshes into clean artist meshes in only 3 seconds, with the lowest Chamfer distance (0.020) and the highest normal consistency (0.860) among all methods. Robustness varies widely: Meshy T2, MeshFlow, MeshAnything V2, and FastMesh all succeed on 100% of test assets, while DeepMesh and MeshSilksong fail to produce valid outputs for more than half of them within the 20-minute limit.

Native image-to-mesh diffusion methods achieve perfect success rates and faster generation than autoregressive two-stage pipelines, with Meshy T2 leading in both speed (6 seconds) and perceptual alignment. MeshFlow yields the closest photometric match but is slower, while DeepMesh and MeshSilksong suffer from low success and very long runtimes, making them impractical for interactive use. Diffusion methods (Meshy T2, Tripo P1, MeshFlow) all reach 100% success, with Meshy T2 and Tripo P1 requiring only 6 and 12 seconds median time. MeshFlow records the lowest Inception FD (254.06), while Meshy T2 achieves the lowest DINOv2 FD (2312.01), indicating stronger semantic alignment with the source photograph. Autoregressive baselines (MeshAnything V2, BPT) are reliable (95.7–100% success) but need 49–210 seconds, an order of magnitude slower than the fastest diffusion method. DeepMesh and MeshSilksong exhibit the highest FD scores and fall below 50% success, with median runtimes exceeding ten minutes, disqualifying them for interactive asset creation. FastMesh sits between the best autoregressive and diffusion methods on DINOv2 FD (2405.57), while DeepMesh deviates severely (2752.54) from the reference distribution.

Ablation on the vertex-set VAE confirms that incorporating optimal transport with Sobol sampling dramatically improves geometric accuracy over simpler ordering or no transport. In high-poly retopology, Meshy T2 achieves the best fidelity and topology in just 3 seconds, while competing methods are either over ten times slower or fail on many assets. For image-to-mesh generation, native diffusion methods like Meshy T2 reach 100% success with fast, perceptually aligned outputs, whereas autoregressive pipelines are slower and some alternatives prove unreliable.