The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos

Key insights

5 slides

New resource

The NES-VMDB dataset pairs gameplay clips with MIDI soundtracks.

The NES Video-Music Database (NES-VMDB) contains 98,940 gameplay clips from 389 Nintendo Entertainment System (NES) games. Each clip is matched with its original soundtrack in MIDI format.

Dataset purpose. The NES-VMDB dataset provides 98,940 gameplay clips from 389 NES games, paired with their original MIDI soundtracks to support generative music model development.

Automated matching

An audio fingerprinting algorithm matched video segments with MIDI files.

Researchers used a Dejavu-based audio fingerprinting algorithm to automatically pair 15-second video clips with their respective synthesized MIDI files.

Like Shazam identifying a song, this algorithm identifies music within gameplay videos.

Matching method. Researchers utilized a Dejavu-based audio fingerprinting algorithm to automatically match video segments with MIDI files, ensuring robustness against game-specific sound effects and noise.

Classification accuracy

A genre classifier achieved 29% accuracy, indicating correlation.

A genre classifier trained on the dataset achieved a 29% accuracy rate, significantly outperforming random guessing.

Correlation found. A genre classifier trained on the dataset achieved 29% accuracy, proving a measurable correlation between gameplay visuals and musical style, though it remains below human-level performance.

1 / 5

Ask AI about this report

Summary

The NES Video-Music Database (NES-VMDB) is a novel dataset designed to facilitate the development of generative models capable of composing symbolic background music based on gameplay video input. By pairing 98,940 gameplay clips from 389 Nintendo Entertainment System (NES) games with their corresponding original soundtracks in MIDI format, the dataset aims to democratize music production for indie game developers. The project builds upon the existing Nintendo Entertainment System Music Database (NES-MDB), which contains 5,278 music pieces, by associating 4,070 of these compositions with 474 hours of gameplay footage.

The methodology involved collecting long-play videos from YouTube and segmenting them into 15-second clips. An audio fingerprinting algorithm, based on the Shazam-style Dejavu implementation, was utilized to automatically match these video clips with their respective synthesized MIDI files. This automated approach was necessary to handle the large volume of data, and testing confirmed the algorithm’s robustness against background noise and sound effects.

To validate the utility of the dataset, a baseline model was established using a Controllable Music Transformer (CMT). Objective metrics—including pitch class histogram entropy, grooving pattern similarity, and pitch range—demonstrated that the conditioned CMT model produces musical structures more closely aligned with human-composed pieces than an unconditional baseline. Additionally, a genre classifier was trained to evaluate the correlation between gameplay visuals and musical output. While the classifier achieved a 29% accuracy rate, significantly outperforming random guessing, the findings indicate that further research is required to achieve human-level performance in both music generation and genre classification. The dataset is intended to serve as a resource for creating tools that allow developers to generate and edit background music for various stages of game development.