“By using state-of-the-art methods and hardware acceleration (a major part of the calculations run on the device’s GPU), we succeeded in enabling the engine to run in real time with unnoticeable latency between the moment the note is played and the identification, even within the limited resources of tablet devices,” said Kaminka. According to the company, the engine framework can be trained to look for specific patterns that match specific musical instruments. “Every piano or keyboard also has its own ‘color’ to the sound, and the MusicSense engine picks up on these, and learns them on the fly providing more accurate feedback,” he continued.
Jamstar, an iOS app that teaches guitar acoustically, takes a similar approach, detecting notes played via the device microphone. The commercial approach is for users to make in-app purchases of songs they’d like to learn on guitar. The visual metaphor is again of scrolling notes that hit a bar indicating when they should be played, while another part of the screen displays the fretboard fingering. Both Jamstar and JoyTunes are surprisingly intuitive and addictive, this reporter found.
Build a better instrument
While pressing the four buttons on a Guitar Hero controller won’t teach fretboard skills (or even build much-needed calluses), another product aims to combine the flashiness of interactive, full-sized instruments with the power of iOS apps. A US$353,392 Kickstarter campaign in 2013 launched San Francisco-based Incident’s gTar, a solid-body electric guitar with LEDs embedded in its fretboard.
A rectangular dock allows an iPhone to be set flush into the front of the guitar body. Running the gTar app, the iPhone detects the player’s actions via MIDI, not acoustically. Similar to JoyTunes and Jamstar, the iPhone screen animates the required notes of a song, displaying them as colored fret positions.
The potential for the instrument is as an educational platform, not just a tool, said Josh Stansfield, director of brand marketing for Incident. “It’s MIDI with more. It can detect fret up and fret down separate from MIDI—things like tapping notes on a fret. We’re adding to it all the time,” he said.
Trying to crack the practice code is still at the heart of the gTar effort. The company hired a handful of non-guitar players via Craigslist to spend an hour or two with the device. “Every person left being able to play at least a section of a song proficiently,” Stansfield claimed.