One-sentence Summary

City University of Hong Kong and HKUST propose DrawingSpinUp, a system that generates 3D animations from single character drawings by adopting a removal-then-restoration strategy to handle view-dependent contour lines and a skeleton-based thinning deformation algorithm to refine slim structures, outperforming existing 2D and 3D animation methods.

Key Contributions

• The paper introduces DrawingSpinUp, a system that reconstructs a 3D character proxy from a single drawing and retargets 3D motion data to produce plausible animated characters. It incorporates a pre-trained image-to-3D diffusion prior to compensate for unseen texture regions, and evaluations plus a perceptual user study show stronger 3D animation quality than existing 2D and 3D animation methods.

• The method contributes a removal-then-restoration appearance pipeline that first removes view-dependent contour lines before reconstruction and then uses a geometry-aware stylization network to render them back after retargeting. This preserves the contour style of the input drawing while avoiding the geometry and texture ambiguity caused by contours.

• The work adds a skeleton-based cutting-and-thinning deformation algorithm to refine slim structures represented by single-line contours, such as thin limbs in stick-figure drawings. Experimental evaluations and the user study indicate that this geometry refinement helps generate more plausible animated characters from simple line-based inputs.

Introduction

Character drawing animation matters for storytelling, gaming, and animation because it lets professionals and amateurs turn a single imagined character sketch into a moving performance. Prior 2D methods can only deform a drawing in the image plane and struggle with true 3D motions, while image to 3D methods are often trained on photo realistic images and misinterpret artistic contour lines as texture details, which can weaken geometry and appearance. The authors leverage a removal then restoration strategy: they first remove view dependent contour lines, reconstruct a textured 3D proxy with a pretrained image to 3D model, refine slim structures with a skeleton based thinning method, then render stylized contours and textures after 3D motion retargeting.

Dataset

• The authors use the 3DBiCar dataset as the training source. The data are created from textured 3D biped cartoon characters rendered with Blender.

• Each training sample is generated from a front-view render of a character. Contour lines with different thicknesses are rendered and then stylized with random colors before being composited onto the front-view images.

• The random colors and varying contour thicknesses are used to imitate amateur drawing styles.

• The authors treat the rendered drawings as training examples for contour prediction. The task is framed as image-to-image translation: given an input drawing and its segmented foreground mask, the model predicts a corresponding contour mask.

• The excerpt does not report the dataset size, train/validation/test split, mixture ratios, filtering rules, or additional metadata fields.

• For contour removal, the predicted contour mask is combined with the background region to form the inpainting region. Pixels inside this region are filled from neighboring visible pixels, while the foreground area is preserved. This step describes how the rendered contour data are used in the contour-removal pipeline.

Method

The authors leverage DrawingSpinUp as a multi-stage system that converts a single character drawing into a stylized 3D animation. Given an input drawing, a foreground segmentation mask, predicted joint keypoints, and a target 3D motion, the pipeline first removes artistic contours to obtain a cleaner interior texture, reconstructs a coarse 3D geometry, refines the geometry and texture, rigs the character for animation, and finally restores the original drawing style on every rendered frame. Refer to the framework diagram:

The first module addresses contour removal and interior inpainting. Because stroke thickness, color, and style can vary across drawings and even within one drawing, the system avoids fixed geometric contour extraction and instead predicts contours through image-to-image translation. Let $I$I be the input drawing and $M$M be its foreground mask. An FFC-ResNet generator predicts a contour mask $M_{c}$Mc​. Fast Fourier convolution is used because its large receptive field helps the network capture object-boundary cues and contour-like structures. The training process synthesizes drawings by rendering front-view images from textured cartoon characters and overlaying stylized contour lines with varying colors and thicknesses.

After predicting $M_{c}$Mc​, the method extends the mask with the background region so that white background pixels do not corrupt the interior reconstruction:

The inpainting region is then filled from neighboring interior pixels using a fast marching method, and the result is projected back to the foreground:

Refer to the contour-removal flowchart:

The contour-free drawing $I_{\mathrm{inpaint}}$Iinpaint​ is then used for coarse 3D reconstruction. The authors apply a pretrained diffusion model to generate multi-view normal maps and color images, which are segmented to obtain view-specific foreground masks. A neural surface reconstructor then converts these 2D representations into a textured 3D geometry. This coarse reconstruction is useful but imperfect: thin strokes can become overly thick, textures may be blurry, and surface adhesion can appear between nearby limbs or body parts. As shown in the surface adhesion example:

To recover the intended front-view silhouette, the method performs front-view cutting on the reconstructed signed distance field. Let $f(\cdot)$f(⋅) be the SDF and $(X,Y)$(X,Y) be the projected 2D coordinates of a 3D point $(x,y,z)$(x,y,z) on the front-view plane. The trimmed geometry is defined by the 0-level set:

This trimmed level set is converted into a mesh using Marching Cubes. However, front-view cutting only constrains the frontal silhouette and does not sufficiently reduce the side thickness of limbs, hair, or other elongated parts.

The system therefore introduces a skeleton-based thinning deformation. Given a trimmed mesh with vertices $\mathbf{v}$v and faces $\mathbf{f}$f, the new vertices are written as:

where $\mathbf{h}$h denotes handle vertices, $\mathbf{d_h}$dh​ denotes known displacements for those handles, and $\mathcal{B}(\cdot)$B(⋅) computes a bi-harmonic deformation field using a uniform Laplacian operator. To determine which vertices should be fixed and which should move, the method extracts a distance map $D$D and a skeleton $S$S from the foreground mask $M$M. It then constructs fixed and movable masks as:

where $\theta_1 > \theta_2$θ1​>θ2​ are distance thresholds. The operation $\mathrm{IR}(\cdot)$IR(⋅) removes pixels in the intersection region between movable skeleton parts and fixed regions, preventing thinning deformation from damaging nearby stable structures. Refer to the handle masking illustration:

The fixed and movable vertex sets are projected from these masks. For movable vertices, the desired displacement is queried from the distance map $D$D, and the thinning deformation is applied. Sharp edges created by cutting and thinning are then smoothed with Laplacian smoothing.

After geometry refinement, texture quality is improved by back-projecting multi-view color images onto the 3D vertices. Since the characters are assumed to be in a forward A-pose or T-like stance, most surface regions can be colored by querying front and back views. Regions that are not directly visible, such as inner limb boundaries, are filled using weighted colors from neighboring vertices. After front-view cutting, skeleton-based thinning, and color back-projection, the geometry becomes substantially closer to the original drawing. Refer to the stage comparison:

The refined contour-free 3D character is then rigged and animated. An automatic rigging process constructs a skeleton from the predicted 2D joint keypoints, making the mesh animation-ready. Skinning weights are computed from the distance between each vertex and the nearest bone. A humanoid motion sequence is retargeted onto the rigged character, producing rendered color frames $\mathcal{F}=\{F_1,F_2,\ldots,F_N\}$F={F1​,F2​,…,FN​}.

The final module restores the artistic style of the original drawing. The stylization network is designed as a two-stage geometry-aware image-to-image translation model. The first modified U-Net, $U_{\text{texture}}$Utexture​, restores internal texture details, while the second modified U-Net, $U_{\text{contour}}$Ucontour​, restores external contour lines. To reduce sensitivity to rotation during dynamic poses, most convolutional layers in $U_{\text{texture}}$Utexture​, except the final layer, are replaced with rotation-invariant coordinate convolutional layers.

The stylization stage uses four guidance channels: the rendered color frame $F$F, a foreground mask $G_{\mathrm{mask}}$Gmask​, a positional hint $G_{\mathrm{pos}}$Gpos​, and an edge map $G_{\mathrm{edge}}$Gedge​. The positional hint is derived from normalized coordinates in a rest pose and encourages texture transfer to consistent semantic locations across views. The edge map is extracted from the Z-depth using a Canny edge detector and provides view-dependent structure for contour restoration. The first network processes $(F,G_{\mathrm{mask}},G_{\mathrm{pos}})$(F,Gmask​,Gpos​) to produce an intermediate texture-restored frame $O'$O′, and the second network processes $(O' + G_{\mathrm{edge}}, G_{\mathrm{mask}}, G_{\mathrm{pos}})$(O′+Gedge​,Gmask​,Gpos​) to produce the final stylized frame $O$O.

Because artistic styles vary widely in color, stroke width, and contour topology, the stylization networks are trained with a patch-based supervised strategy. Random patches are sampled from all guidance channels and the ground-truth stylized images. These patch pairs supervise $U_{\text{texture}}$Utexture​ to recover interior textures and $U_{\text{contour}}$Ucontour​ to recover stylized contours. The training objective combines L1 loss, adversarial loss, and VGG loss. The loss between $O'$O′ and $I_{\mathrm{inpaint}}$Iinpaint​ optimizes texture restoration, while the loss between $O$O and the original drawing $I$I optimizes contour restoration. At inference time, the fully convolutional architecture allows full-frame inputs to generate the final stylized animation sequence.

Experiment

Experiments evaluate efficiency, animation quality, and design choices on a single RTX 4090, showing that per-character modeling and stylization training are performed once and then support fast repeated animation generation. Qualitative comparisons indicate that the method produces more faithful 3D-aware motion than a 2D projection baseline and avoids the texture artifacts common with general image-to-3D reconstruction backbones. User study results and ablations further support the importance of contour removal, cutting-and-thinning, and rotation-invariant convolution for preserving input style, handling slim structures, and maintaining stable, view-consistent results.