Gamifying Data Collection: A Video Game to Teach AI How Music Makes You Feel

Could you briefly introduce your thesis? What is it specifically about?

My project explores the intersection of AI, music, and emotion. Emerging applications in artificial intelligence rely on data that captures how music interacts with human emotions. Examples include affective music generation (AI-AMG), emotion recognition systems, and music information retrieval (MIR).

This data is usually collected through questionnaires, web scraping, or crowdsourcing. Given the complexity of emotions and the time-consuming nature of obtaining labels, most existing datasets are too small or noisy. Moreover, on-demand tasks do not necessarily evoke the same reactions as spontaneous events. Asking to observe emotions could unintentionally alter them, which results in data that fails to reflect natural listening experiences.

To address this, I created Symphony of Adventure – a role-playing video game that embeds data collection. Instead of filling in forms, players provide labels implicitly through gameplay actions. This approach aims to simulate real-life scenarios where people naturally associate music with moods, in hopes of improving ecological validity and reducing the bias of introspection.

The data collected classifies music based on emotional content and can serve as ground truth for machine learning tasks, or for studying the relationship between emotion and music in general.

What inspired you to focus on this topic?

When choosing my thesis topic, I knew it had to be something I was genuinely interested in, so that I could stay motivated alongside many other courses and obligations. I asked myself how I could combine my other passions (music and video games) with computer science to create a project that would keep me invested. My initial idea was to study the impact of music on player engagement, but with the help of my supervisor, MgA. Jan Hajič, Ph.D., I learned the importance of collecting reliable emotion-related data for AI research.

Can you explain the specific benefits or uses of your work?

The main purpose of this project is to provide researchers with a tool they can adapt using their own music sets. During gameplay, users’ responses are automatically exported, turning the project into a practical system for collecting emotional annotations in music for any research purpose.

Compared to traditional approaches, the benefits we aim for are:

• Improved ecological validity: the game tries to simulate natural listening environments.
• Reduction of the cognitive load: annotation is integrated into gameplay.
• Wider reach: the game itself serves as an incentive for participation.

Additionally, I hope it demonstrates how gamification can make annotation tasks, which are an essential part of artificial intelligence, less tedious and more enjoyable.

What technologies have you worked with, what methods have you used, and why these?

The project was developed in C# using the Unity engine. This game engine was selected since it is a well-known, reliable development tool that is used in big commercial settings as well as indie (smaller-scale) settings.

For narrative design, I used Yarn Spinner, which simplified the creation of branching dialogues.

To manage collected data and maximize the usability of this tool for the layman, I integrated the Google Sheets API. This enables researchers to import metadata and automatically export players’ responses in real time. I added this so the data is organized and ready for analysis.

To optimize performance, I used Unity’s Addressables system to load music at runtime from a (possibly large) set based on a configuration file. This avoids unnecessarily large builds. I also developed custom solutions, such as persistent Scriptable Objects for state preservation, to meet the project’s unique requirements.

What was the most challenging part of writing your thesis? Was there something you got stuck on, some path that didn't lead anywhere? Is there anything you would have done differently in hindsight?

The biggest challenge was balancing two goals: creating an engaging game while maintaining scientific rigor in the emotion models and annotation process. Every narrative element (characters, missions, and dialogues) had to be designed to reflect a carefully chosen model of emotions rather than unadulterated imagination. This was essential, since obtaining meaningful results relies on a systematic and empirical approach to measuring emotions.

I also faced the typical challenges of solo game development, which involve multi-domain work: programming, graphical design, sound design, narrative design, dialogue design, physics design, quality assurance, and optimization. Making sure none of these were getting overlooked was difficult.

On the technical side, integrating Google authentication for secure data export was particularly complex; each instance of the game provides data and needs access to a Google sheet that does not belong to the user. The only viable solution was to use a Google Service account, but service account keys pose security risks if not managed properly. I implemented encryption, which added more unexpected work.

How did you verify the results of your work?

I ran several testing iterations with colleagues, friends, and family by having them play and answer surveys. It provided insight to (a) how attractive and interesting the game is and (b) how precise and effective the data collection mechanism is.

This cycle of developing the game, presenting its current state to testers, receiving feedback, applying the feedback, and solving bugs was an inseparable part of the work.

The Prague Music Computing Group (PMCG), a new research group focused on computational processing and modelling of music, was an invaluable resource. It allowed me to verify the validity of the concept and receive feedback from specialists and students in the field.

What do you consider to be the most crucial result or conclusion of your work?

This project provides a proof of concept that collecting emotional annotations through a game is feasible. It demonstrates a creative and practical alternative to traditional annotation methods, which are often expensive, noisy, and lacking in ecological validity.

Do you feel your work can inspire other students or professionals in the field?

This project shows you can combine your passions, even ones that seem unrelated, and create something new. I hope it encourages students to think outside the box about how computer science can intersect with music, or any other field they care about.

Additionally, I hope this makes people see the potential in gamification as a valuable tool in research, and not just entertainment.

What are your plans for the future?

I would like to continue working at the intersection of AI, video games, and music. The next step for this project is to use the tool for its intended purpose – to gather a dataset and train an affective music generation model. My vision is to release it as a Unity Asset Package for indie game developers, who often lack the resources to create emotion-driven music.

Why did you decide to study at the Faculty of Mathematics and Physics? Would you recommend it to other English-speaking students?

I decided to study at MATFYZ to fulfil my dream of living and studying abroad, after discovering my love for computer science in high school. After examining a few options around Europe, I chose this program due to the faculty’s strong reputation, the affordability of living in Prague, the hospitality of the Czech people towards foreigners, and a beautiful first impression the city left on me during a visit.

I would recommend the programme to English-speaking students seeking a challenging but rewarding experience. The faculty offers a chance to fully dedicate yourself to a subject and grow quickly.

For me, it also provided the invaluable opportunity to join the Prague Music Computing Group — something I never imagined possible when I thought my interest in music and AI was too niche. Being surrounded by people who share this passion and bring so much knowledge has been an inspiring experience. This is thanks to the great initiative of my advisor MgA. Jan Hajič, PhD who formed the group.

Links

GitHub – Symphony of Adventure
Prague Music Computing Group
University repository