SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches

The research introduces SketchBetween, a machine learning system designed to automate the labor-intensive process of 2D sprite animation in game development. The primary thesis is that existing video-to-video synthesis methods, which typically focus on real-life footage, fail to account for the stylistic, exaggerated motion inherent in 2D animation. By utilizing a vector-quantized variational auto-encoder (VQ-VAE), the system maps fully rendered start and end keyframes, combined with artist-provided sketches of intermediate frames, to generate high-quality, stylized sprite animations.

The model was trained and evaluated using the MGIF dataset, which contains 1,000 videos of cartoon animals in motion. The methodology involved preprocessing these videos by generating sketches via Canny edge detection to simulate an artist’s workflow. Quantitative performance was measured using the structural similarity index metric (SSIM) and peak signal-to-noise ratio (PSNR). In comparative testing, the system outperformed a baseline adapted from a first-order motion model (FOMM), achieving an SSIM of 0.943 compared to the baseline’s 0.799. An ablation study confirmed that the inclusion of intermediate sketches significantly improves output quality, validating the proposed animation workflow.

The scope of the research covers 2D sprite animation, specifically targeting game development assets like character walk and jump cycles. While the model demonstrates success in capturing stylized motion, the study notes limitations regarding animations with drastic, non-linear movements and a tendency for output to appear blurry. Future development aims to explore diffusion-based synthesis and more iterative design tools to further reduce the time and technical skill required for professional-grade 2D animation.