TouchTones, created by our crew Stimulant, is an interactive, multi-user, multi-touch music maker for Microsoft Surface.

Inspired by the work of Toshio Iwai and originally conceived (and entirely developed) by  the insanely talented Josh Santangelo, I led the creative direction and interaction design, and I also created all the sounds for the piece. Our goal in making TouchTones was to ensure that anyone could use it with only a few seconds of exploration, and create beautiful music without any musical training. It was all about immediacy and richness, and the sound needed to support this.

TouchTones is a grid-based music sequencer: the user sets a sprite in motion that, when passing over a grid node, makes a specific sound. Each sprite is a different instrument, moving at different speeds, but are all locked to a master tempo. There are four sprites (voices) and 32 nodes (pitches/notes).

The main challenge was placing notes on the grid. I started by composing short pieces of music that featured a lot of arpeggios of varying note durations, which mimicked how the nodes on the grid would get triggered. This helped me figure out the best note durations for certain sounds, and to establish a key to work in. Since the user is the one who creates the final melody, the only way to really stress-test the sounds and key was to prototype and have real people play with it.

The sound palette itself went through several iterations. The first featured somewhat realistic sounds with a pretty complex scale, so the likelihood of atonality was too high. The second iteration featured purely electronic sounds in a more harmonious scale, but the sounds were too aggressive (probably owing to my own past attraction towards angry music). The third and final iteration finally hit the mark: Cleaner, primarily acoustic sounds, a key that’s pleasant and even a bit wistful, and a note distribution that isn’t always linear, preventing unnatural shifts into inappropriate pitch registers. Internally, we jokingly call the final result the “indie film about autumn in Central Park” palette.

All the sounds were created in Logic Pro, primarily using the EXS24 sampler. A lot of tonal and envelope tweaking ensued. Rather than provide sound clips like I usually do, I encourage you to watch the embedded video above to get a sense of how the application feels and sounds.

Microsoft hired Stimulant to help convey the breadth and depth of their corporate social responsibility efforts; we designed an interactive online map that was cool, but we followed that up with a version for Microsoft Surface. I led the interaction design effort for both projects. This Local Impact Map, as it was called, was intended to show the positive human and economic impacts of Microsft’s global citizenship efforts.

When we created the Surface version, the interface’s lack of sound became glaringly apparent. As with any project, even for an outspoken advocate of sound like myself, audio often comes last when things get super busy. Surface is a highly sonified platform, though, with outstanding sound design. “A silent Surface app is a dead Surface app,” says fellow Surface developer Infusion. Too true – in fact, our first Surface application was a music sequencer. So, I set about trying to think about what sonic palette would be appropriate.

That was the wrong thing to do, actually. I stared, thought, listened, sketched. No single set of sounds came to mind.

Instead, I finally had a conceptual breakthrough: Rather than figure out how the application should sound, I decided to focus instead on how those sounds would be made. Given the message and brand, I decided that all the sounds had to originate with simple objects and instruments that are manipulated by human hands. This seemed to get closer to the organic and directly-human heart of the project’s message.

I arrived at an odd set of objects that human hands could make cool sounds with. I whittled these down to only two objects: a kalimba and a one liter water bottle. The kalimba, of course, was played somewhat normally, but the one liter water bottle was tied to a string and swung by a microphone dozens of times, clapped, crunched, and blown upon. From all of these samples came a fairly small and concentrated set of sounds for positive feedback, errors, and transitions. I altered the volume envelopes on most of these sounds to make them either more pronounced or less percussive, and then applied some equalization and compression to make them all fit together, especially when played together…this is a multi-user application, after all.

Since the video above shows the map being used but doesn’t feature the interface’s audio, here’s a compilation of the actual sounds for your listening pleasure.

Sound Design of the Microsoft Local Impact Map for Surface by noisejockey
[Røde NT1a mic into Sound Devices 702 recorder, Soundtrack Pro and Peak Pro for post-processing]