(Related: What gamification means to developers)
The hand-eye-ear coordination of gaming is potentially analogous to music, if making sounds on an instrument can be made technically easier. Where JoyTunes shows tremendous promise is in using the form factor of the real instrument rather than a shoddy-sounding replica. Giving children a game-like interface to practice songs has been a critical and market success thus far.
“JoyTunes has developed ways to start experiencing the ‘music-derived dopamine effect’ right from the start,” said Kaminka. “Orchestral/band accompaniment from the very first note to let players feel they are always taking part in music creation. Simplified versions of well-known songs (from classical to pop). And e-mails to parents, keeping them always in the loop of the child’s studies. In other words, creating an audience.”
In “The Power of Habit,” author Charles Duhigg shows that removing even the simplest barriers to practicing (such as keeping the instrument stored in a closet) is critical to forming a positive new habit. Thus, delivering a music-teaching app on a widely available mobile platform makes sense for music practice, which typically occurs away from offices or desktop computers. But what about making the app cable-free? That, too, is becoming widespread as signal processing has matured.
Audio fingerprinting was famously commercialized by Shazam Entertainment, which launched in 2000 with a mobile phone-based music recognition service. “The algorithm had to be able to recognize a short audio sample of music that had been broadcast, mixed with heavy ambient noise, subject to reverb and other processing, captured by a little cellphone microphone, subjected to voice codec compression and network dropouts, all before arriving at our servers,” according to Shazam scientist and cofounder Avery Li-Chun Wang in “An Industrial-Strength Audio Search Algorithm.” “The algorithm also had to perform the recognition quickly over a large database of music with nearly 2M tracks, and furthermore have a low number of false positives while having a high recognition rate.”
More than a decade later, a number of apps are able to do the tone recognition that Shazam pioneered. In terms of pure music recognition via combinatorially hashed time-frequency constellation analysis of the audio, alternatives to Shazam include SoundHound, musiXmatch, Midomi and Tunatic. There are at least two open-source projects (MusicBrainz and Echoprint, which uses The Echo Nest’s database), which together allow anyone to build music fingerprinting into their application. That’s a slightly different task than what music education apps must do: Check that the right notes are played in tune and on time.
“Shazam relies on server matching: They collect a series of sounds throughout the track and send them to their server for matching, whereas what our MusicSense Engine does is instantaneously identify that a note was played, confirm that it was from the musical instrument and not part of background noise, and decide which note it was in order to provide feedback,” said Sivan Finn, VP of marketing for JoyTunes.
“All of this needs to be performed instantaneously and on the client-side rather than server-side.”
According to PatentBuddy.com, JoyTunes has a patent pending for “a system and method for improving musical education through use of a game” which includes “receiving electrical signals associated with a musical piece provided by a user of the game,” applying a discrete Fourier transform and further analysis. The end result is real-time polyphonic transcription of musical notes played by any instrument or voice.