Forget what you’ve heard about the music business. While it may be hard to feed a family of four on gigs, the market for musical gear and education is exploding. In 2014, new instruments and apps for learning how to play them are taking advantage of software and embedded technology. Take the fantastical circular keyboard called PianoArc.
The donut-shaped keyboard is big enough for two grown men to stand inside it, back to back, and play simultaneously. It is enabled by an optical key detection system called PNOscan, the latest from QRS, a 114-year-old American company that once supplied music rolls to the player piano market.
The instrument is likely just a novelty for touring acts like Lady Gaga (or so it advertises). For music teachers, new ways of integrating computers and acoustic instruments abound. Piano Marvel can run on a touch-screen tablet or laptop sitting on a piano’s music stand. Music notation scrolls by on a white background as you play the piano.
“This is music education software. Most people use a laptop plugged into their electronic keyboard,” said Joel Shifflet, product manager at Piano Marvel. Pairing Piano Marvel with an acoustic instrument requires retrofitting the piano with PNOscan optical sensors so that a MIDI (musical instrument digital interface) cable can send the note output to a computer.
The system itself, however, marks one example of a growing number of apps designed to add “gamification” and “badgification” goodies to music education software. The advantage of Piano Marvel is that advanced students can track the accuracy of their sight-reading without any signal processing latency. They can also post scores to a cloud server and keep their teachers informed on progress. Students earn trophies and keep them in a virtual case. No sound effects or bright Guitar Hero-like flashes are to be found.
“We looked at adding more gamification aspects, but we found them distracting to students,” Shifflet said.
Make it fun and it gets done
But as a glance at mobile app stores will show you, gamification is in full force. PianoMaestro was launched in 2013 (as Piano Mania), one of a family of three iOS music education apps by the Israeli startup. Another, Piano Dust Buster, uses the same falling notes metaphor that Guitar Hero made famous, itself building on the dopamine-inducing effect of fitting falling Tetris blocks together.
“JoyTunes was inspired by two main research directions,” said Yigal Kaminka, musical director and cofounder of JoyTunes. “First, the extensive research around the benefits of gamificationin education in general. Second was a big survey conducted in the U.S. in 2010 showing that nearly 85% of those who start learning to play a musical instrument quit during the first one to two years. At this time, gamification in music education was still very new and undeveloped, and it was clear to us that in order to combat these terrible figures we would have to develop and use gamification as a pedagogical tool.”
(Related: What gamification means to developers)
The hand-eye-ear coordination of gaming is potentially analogous to music, if making sounds on an instrument can be made technically easier. Where JoyTunes shows tremendous promise is in using the form factor of the real instrument rather than a shoddy-sounding replica. Giving children a game-like interface to practice songs has been a critical and market success thus far.
“JoyTunes has developed ways to start experiencing the ‘music-derived dopamine effect’ right from the start,” said Kaminka. “Orchestral/band accompaniment from the very first note to let players feel they are always taking part in music creation. Simplified versions of well-known songs (from classical to pop). And e-mails to parents, keeping them always in the loop of the child’s studies. In other words, creating an audience.”
Shazam this!
In “The Power of Habit,” author Charles Duhigg shows that removing even the simplest barriers to practicing (such as keeping the instrument stored in a closet) is critical to forming a positive new habit. Thus, delivering a music-teaching app on a widely available mobile platform makes sense for music practice, which typically occurs away from offices or desktop computers. But what about making the app cable-free? That, too, is becoming widespread as signal processing has matured.
Audio fingerprinting was famously commercialized by Shazam Entertainment, which launched in 2000 with a mobile phone-based music recognition service. “The algorithm had to be able to recognize a short audio sample of music that had been broadcast, mixed with heavy ambient noise, subject to reverb and other processing, captured by a little cellphone microphone, subjected to voice codec compression and network dropouts, all before arriving at our servers,” according to Shazam scientist and cofounder Avery Li-Chun Wang in “An Industrial-Strength Audio Search Algorithm.” “The algorithm also had to perform the recognition quickly over a large database of music with nearly 2M tracks, and furthermore have a low number of false positives while having a high recognition rate.”
More than a decade later, a number of apps are able to do the tone recognition that Shazam pioneered. In terms of pure music recognition via combinatorially hashed time-frequency constellation analysis of the audio, alternatives to Shazam include SoundHound, musiXmatch, Midomi and Tunatic. There are at least two open-source projects (MusicBrainz and Echoprint, which uses The Echo Nest’s database), which together allow anyone to build music fingerprinting into their application. That’s a slightly different task than what music education apps must do: Check that the right notes are played in tune and on time.
“Shazam relies on server matching: They collect a series of sounds throughout the track and send them to their server for matching, whereas what our MusicSense Engine does is instantaneously identify that a note was played, confirm that it was from the musical instrument and not part of background noise, and decide which note it was in order to provide feedback,” said Sivan Finn, VP of marketing for JoyTunes.
“All of this needs to be performed instantaneously and on the client-side rather than server-side.”
According to PatentBuddy.com, JoyTunes has a patent pending for “a system and method for improving musical education through use of a game” which includes “receiving electrical signals associated with a musical piece provided by a user of the game,” applying a discrete Fourier transform and further analysis. The end result is real-time polyphonic transcription of musical notes played by any instrument or voice.
“By using state-of-the-art methods and hardware acceleration (a major part of the calculations run on the device’s GPU), we succeeded in enabling the engine to run in real time with unnoticeable latency between the moment the note is played and the identification, even within the limited resources of tablet devices,” said Kaminka. According to the company, the engine framework can be trained to look for specific patterns that match specific musical instruments. “Every piano or keyboard also has its own ‘color’ to the sound, and the MusicSense engine picks up on these, and learns them on the fly providing more accurate feedback,” he continued.
Jamstar, an iOS app that teaches guitar acoustically, takes a similar approach, detecting notes played via the device microphone. The commercial approach is for users to make in-app purchases of songs they’d like to learn on guitar. The visual metaphor is again of scrolling notes that hit a bar indicating when they should be played, while another part of the screen displays the fretboard fingering. Both Jamstar and JoyTunes are surprisingly intuitive and addictive, this reporter found.
Build a better instrument
While pressing the four buttons on a Guitar Hero controller won’t teach fretboard skills (or even build much-needed calluses), another product aims to combine the flashiness of interactive, full-sized instruments with the power of iOS apps. A US$353,392 Kickstarter campaign in 2013 launched San Francisco-based Incident’s gTar, a solid-body electric guitar with LEDs embedded in its fretboard.
The gTar in action.
A rectangular dock allows an iPhone to be set flush into the front of the guitar body. Running the gTar app, the iPhone detects the player’s actions via MIDI, not acoustically. Similar to JoyTunes and Jamstar, the iPhone screen animates the required notes of a song, displaying them as colored fret positions.
The potential for the instrument is as an educational platform, not just a tool, said Josh Stansfield, director of brand marketing for Incident. “It’s MIDI with more. It can detect fret up and fret down separate from MIDI—things like tapping notes on a fret. We’re adding to it all the time,” he said.
Two SDKs, one in Objective-C andone in JavaScript, allow prospective app builders, instrument hackers, musicians or teachers to toy with the inputs and outputs of the guitar.
“The reason we put JavaScript in is to do Web apps, because MIDI is going to be incorporated into Chrome,” said Stansfield.“There are plug-ins available now.”
Trying to crack the practice code is still at the heart of the gTar effort. The company hired a handful of non-guitar players via Craigslist to spend an hour or two with the device. “Every person left being able to play at least a section of a song proficiently,” Stansfield claimed.
“We’ve gotten more feedback, and we see that once you get past the initial five to 10 minutes, people want to hit the reset button. Then they say, ‘Now give me a curriculum. How do I learn chord structures?’ So we’re building a platform where it’s a framework for lessons that teachers can provide themselves. We’re crowdsourcing content for it.”
The crowdsource model is one that a slightly older MIDI-based piano practice app, Synthesia, has also enabled. The program can read any MIDI file that’s supplied, turning the composition not only into a pleasant scrolling notational interface, but also the graphic falling note-squares everyone loves.
Better felt than heard?
There are technical tradeoffs to be made, of course. Waiting for a computer to detect notes introduces new latency beyond the time it takes for sound to travel to the human ear (though a 10-millisecond wait isn’t much). However, detecting notes via a wired MIDI connection is inherently faster than via a smartphone microphone. To be sure, there are already instrument pickups on the market that can send acoustic sounds to a computer or digital audio workstation via MIDI.
“Pickups are getting more accurate, but there are still latency issues,” said Stansfield. “They also tend to be pretty expensive.”
Ease of use will reign for those who just want to get started. Beyond that, the decision to be a music maker or a music fan may be a fundamental divide that app developers will have trouble bridging. Similar to unfounded claims that software developers will be made extinct by smart IDEs, it seems even the most spectacular technology doesn’t trigger the desire to make music from scratch in everyone, only a select few.
“How can we get you to that epiphany moment when you’ve played a Beatles song and you’ve only been at it for an hour?” said Stansfield.“We’ve been interviewing musicians, asking them, ‘What was the spark that got you practicing?’ They say, ‘Just the realization that I as a mortal could do this was enough motivation to do this every day.’ Not everyone is going to want to do this.”
That’s what Harmonix Music Systems, the creators of Guitar Hero and Rock Band, discovered with their music-making efforts in the late 1990s. Without hit songs, consumers lost interest. It was the karaoke effect that founders Alex Rigopulos and Eran Egozy observed in Japan prior to success with music games. However, Harmonix is poised to revisit the full-body controls (CamJam) they pioneered nearly 20 years ago in a new Disney game, Fantasia.
It’s about the songs
Even if there is a difference between music consumption and music making, most agree that the basic unit of currency is songs, not notes. Pulling students into a musical experience that blends songs (which trigger both memory and emotion) with achievement(which triggers dopamine), may push music learning to new levels.
“We are constantly updating and improving the JoyTunes curriculum based on feedback we’re getting from over 15,000 registered piano teachers,” said JoyTunes’ Kaminka.“We will be releasing a major update to our curriculum, called the Journey, based on this feedback.
“You have to understand that though we have carefully examined many of the standard traditional piano method books, what we’re creating in the new media platform with all its additional options is something totally new. There isn’t a single method book that enables instant feedback on every note you play, nor is there one that includes integrated, hassle-free orchestral/band accompaniment for every single song and exercise.”
No risk, no reward
At higher levels, some music education apps are focused only on content, not interaction. New Orleans-based Tutti Music Player comes out of the jazz tradition and offers customizable video insight into how each player in an ensemble creates a given groove, be it jazz, salsa, funk or second line.
Most music teaching apps, however, are designed to help students and music teachers make practicing fun. But is there such thing as too much fun?After all, any parent knows that these days most children would gladly play video games for eight hours at a stretch.
“As for possible risks in getting music beginners hooked and possibly losing interest in physical instruments — to be honest, though gamification in education in general is already a well-established trend, gamification (in the sense of video-game-like mechanics) in music education is still a very new field, so we can’t state the exact effect,” said Kaminka. “However, all the evidence clearly points to the opposite. If you take Guitar Hero, for instance, their great success lead to a burst of people that ‘converted’ to go study guitar for real, because the minute you can actually make music on your own, you don’t need the video game anymore.”
“Our bigger concern as music educators should be about what’s happening today, as the traditional music education world is fighting for the hearts of music beginners—and losing,” said Stansfield.“The fact is, 85% of all people who start to learn a musical instrument quit in their first year or two.
“I think the idea of interactive instruments that input information back to you, that’s a new concept. We’re seeing that people want more and more stuff like that. There have been instruments with light up keyboards, but not till iPhone could you go both ways.”
In other words, the piano of the future will tell you what you did right—and wrong. Just what the music world needed: more critics!
