A multi-disciplinary designer’s journey in field recording, sound design, and music.

Musical Sound Design for Installations

Posted: April 12th, 2018 | Author: | Filed under: interactive audio, sound design, theory

Want a challenge? Try to play back interface sounds on the show floor at CES. [Intel Booth, CES 2012.]

Want a challenge? Try to play back interface sounds on the show floor at CES. [Intel Booth, CES 2012.]

For those that might not know,  for the last decade I earned my living designing digital installations: Multi-touch interactive walls, interactive projection mapping, gestural interfaces for museum exhibits, that sort of thing. Sometimes these things have sound, other times they don’t. When these digital experiences are sonified, regardless of whether they are imparting information in a corporate lobby or being entertaining inside of a museum, clients always want something musical over something abstract, something tonal over something mechanical or atonal.

In my experience, there are several reasons for this. [All photos in this post are projects that I creative-directed and created sound for while I was the Design Director of Stimulant.]

Expectations and Existing Devices

It’s what people expect out of computing devices. The computing devices that surround me almost all use musical tones for feedback or information, from the Roomba to the XBox to Windows to my microwave. It could be synthesized waveforms or audio-file playback, depending on the device, but the “language” of computing interfaces in the real world have been primarily musical, or at least tonal/chromatic. This winds up being a client expectation, even though the things I design tend not to look like any computer one uses at home or work.

Roomba

Yes, I strapped wireless lavs to my Roomba. The things I do for science.

Devices all around us also use musical tropes for positive and negative message conveyance. From Roombas to Samsung dishwashers, tones rising in pitch within a major key or resolving to a 3rd, 5th, or full octave are used to convey positive status or a message of success. Falling tones within a minor key or resolving to odd intervals are used to convey negative status or a message of failure. These cues, of course, are entirely culture-specific, but they’re used with great frequency.

The only times I’ve ever heard non-musical, actually annoying sound is very much on purpose and always to indicate extremely dire situations. The home fire alarm is maybe the ultimate example, as are klaxons at military or utility installations. Trying to save lives is when you need people’s attention above all else. However, even excessive use of such techniques can lead to change blindness, which is deep topic for another day. Do you really want a nuclear engineer to turn a warning sound off because it triggers too often?

The Problem with Science Fiction

Science fiction interface sounds often don’t translate well into real world usage.

This prototype "factory of the future" had to have its sound design elevated over the sounds of compressors and feeders to ensure zero defects. [GlaxoSmithKline, London, England]

This prototype “factory of the future” had to have its sound design elevated over the sounds of compressors and feeders to ensure zero defects…and had to not annoy machine operators, day in and day out. [GlaxoSmithKline, London, England]

My day job has been inexorably linked to science fiction: The visions of computing devices and interfaces that are shown in films like Blade Runner, The Matrix, Minority Report, Oblivion,  Iron Man, and even televisions shows like CSI set the stage for what our culture (i.e., my client) sees as future-thinking interface design. (There’s even a book about this topic.) People think transparent screens look cool, when in reality they’re a cinematic conceit so that we can see more of the actors, their emotions, and their movement. These are not real devices – they, and the sounds they make, are props to support a story.

Audio for these cinematic interfaces – what Mark Coleran termed FUI, or Fantasy User Interfaces – may be atonal or abstract so that it doesn’t fight with the musical soundtrack of the film. If such designs are musical, they’re more about timbres than pitch, more Autechre than Arvo Part. This just isn’t a consideration in most real-world scenarios.

Listener Fatigue

Digital installations are not always destinations unto themselves. They are often located in places of transition, like lobbies or hallways.

I’ve designed several digital experiences for lobbies, and there’s always one group of stakeholders that I need to be aware of, but my own clients don’t bring to the table: The front desk and/or security staff. They’re the only people who need to live with this thing all day, every day, unlike visitors or other employees who’ll be with a lobby touchwall for only a few moments during the day. Make these lobby workers annoyed and you’ll be guaranteed that all sound will be turned off. They’ll unplug the audio interface from the PC powering the installation, or turn the PC volume to zero.

This lobby installation started with abstract chirps, bloops, and blurps, but became quite musical after the client felt the sci-fi sounds were far too alienating. [Quintiles corporate lobby, Raleigh NC]

This lobby installation started with abstract chirps, bloops, and blurps, but became quite musical after the client felt the sci-fi sounds were far too alienating. Many randomized variations of sounds were created to lessen listener fatigue. There was also one sound channel per screen, across five screens. [Quintiles corporate lobby, Raleigh NC]

Music tends to be less fatiguing than atonal sound effects, in my experience, and triggers parts of the brain that evoke emotions rather than instinctual reactions (in ways that neuroscience is still struggling to understand). But more specifically, sounds with without harsh transients and with relatively slow attacks are more calming.

Randomized and parameterized/procedural sounds really help with listener fatigue as well. If you’re in game audio, the tools used in first- and third-person games to vary footsteps and gunshots are incredibly important to creating everyday sounds that don’t get stale and annoying.

The Environment

Another reality is that our digital experiences are often installed in acoustically bright spaces, and technical sounding effects with sharp transients can really bounce around untreated spaces…especially since many corporate lobbies are multi-story interior atriums! A grab bag of ideas have evolved from years of designing sounds for such environments.

This installation had no sound at all, despite our best attempts and deepest desires. The environment was too tall, too acoustically bright, and too loud. Sometimes it just doesn't work. [Genentech, South San Francisco, CA]

This installation had no sound at all, despite our best attempts and deepest desires. The environment was too tall, too acoustically bright, and too loud. Sometimes it just doesn’t work. [Genentech, South San Francisco, CA]

Many clients ask for directional speakers, which comes with three big caveats. First, they are never as directional as the specification indicate. A few work well, but many don’t, so caveat emptor (they also come with mounting challenges). Second, their frequency response graphs look like broken combs, partially a factor of how they work, and so you can’t expect smooth reproduction of all sound. Finally, most are tuned to the human voice, so of course musical sound reproduction is not only compromised sonically, but anything lower than 1 kHz starts to bleed out of the specified sound cone. That’s physics, anyway – not much will stop low-frequency sound waves except large air gaps with insulation on both sides.

The only consistently effective trick I’ve found for creating sounds that punch through significant background noise is rising or falling pitch, which lends itself nicely to musical tones that ascend or descend. Most background noise tends to be pretty steady-state, so this can help a sound punch through the environmental “mix.”

One cool trick is to sample the room tone and make the sounds in the same key as the ambient fundamental – it might not be a formal scale, but the intervals will literally be in harmony with one another.

Broadband background noise can often mask other sounds, making them harder to hear. In fact, having the audio masked by background noise if you’re not right in front of the installation itself might be a really good idea. I did a corporate lobby project where there was an always-running water feature right behind the installation we created; since it was basically a white noise generator, it completely masked the interface’s audio for passersby, keeping the security desk staff much happier and not being intrusive into the sonic landscape for the casual visitor or the everyday employee.

Music, Music Everywhere

Of course, sometimes an installation is meant to actually create music! This was the first interactive multi-user instrument for Microsoft Surface, a grid sequencer that let up to four people play music.

Of course, sometimes an installation is meant to actually create music! This was the first interactive multi-user instrument for Microsoft Surface, a grid sequencer that let up to four people play music.

These considerations require equal parts composition and sound design, and a pinch of human-centered design and empathy. It’s a fun challenge, different than sound design for traditional linear media, which usually focuses on being strictly representative or on re-contextualized sounds recorded from the real world. Listen to devices around you in real life and see if you notice the frequency (pun intended) with which musical interface sounds are commonplace. If you have experiences and lessons from doing this type of work yourself, please share in the comments below.

Tags: , , , | No Comments »

The Sound Design of TouchTones

Posted: March 12th, 2010 | Author: | Filed under: interactive audio, music, sound design, synthesis

TouchTones, created by our crew Stimulant, is an interactive, multi-user, multi-touch music maker for Microsoft Surface.

Inspired by the work of Toshio Iwai and originally conceived (and entirely developed) by  the insanely talented Josh Santangelo, I led the creative direction and interaction design, and I also created all the sounds for the piece. Our goal in making TouchTones was to ensure that anyone could use it with only a few seconds of exploration, and create beautiful music without any musical training. It was all about immediacy and richness, and the sound needed to support this.

TouchTones is a grid-based music sequencer: the user sets a sprite in motion that, when passing over a grid node, makes a specific sound. Each sprite is a different instrument, moving at different speeds, but are all locked to a master tempo. There are four sprites (voices) and 32 nodes (pitches/notes).

The main challenge was placing notes on the grid. I started by composing short pieces of music that featured a lot of arpeggios of varying note durations, which mimicked how the nodes on the grid would get triggered. This helped me figure out the best note durations for certain sounds, and to establish a key to work in. Since the user is the one who creates the final melody, the only way to really stress-test the sounds and key was to prototype and have real people play with it.

The sound palette itself went through several iterations. The first featured somewhat realistic sounds with a pretty complex scale, so the likelihood of atonality was too high. The second iteration featured purely electronic sounds in a more harmonious scale, but the sounds were too aggressive (probably owing to my own past attraction towards angry music). The third and final iteration finally hit the mark: Cleaner, primarily acoustic sounds, a key that’s pleasant and even a bit wistful, and a note distribution that isn’t always linear, preventing unnatural shifts into inappropriate pitch registers. Internally, we jokingly call the final result the “indie film about autumn in Central Park” palette.

All the sounds were created in Logic Pro, primarily using the EXS24 sampler. A lot of tonal and envelope tweaking ensued. Rather than provide sound clips like I usually do, I encourage you to watch the embedded video above to get a sense of how the application feels and sounds.

Tags: , , , , , , , , | 1 Comment »

The Sound Design of the Microsoft Local Impact Map

Posted: August 15th, 2009 | Author: | Filed under: interactive audio, sound design

Microsoft hired Stimulant to help convey the breadth and depth of their corporate social responsibility efforts; we designed an interactive online map that was cool, but we followed that up with a version for Microsoft Surface. I led the interaction design effort for both projects. This Local Impact Map, as it was called, was intended to show the positive human and economic impacts of Microsft’s global citizenship efforts.

When we created the Surface version, the interface’s lack of sound became glaringly apparent. As with any project, even for an outspoken advocate of sound like myself, audio often comes last when things get super busy. Surface is a highly sonified platform, though, with outstanding sound design. “A silent Surface app is a dead Surface app,” says fellow Surface developer Infusion. Too true – in fact, our first Surface application was a music sequencer. So, I set about trying to think about what sonic palette would be appropriate.

That was the wrong thing to do, actually. I stared, thought, listened, sketched. No single set of sounds came to mind.

Instead, I finally had a conceptual breakthrough: Rather than figure out how the application should sound, I decided to focus instead on how those sounds would be made. Given the message and brand, I decided that all the sounds had to originate with simple objects and instruments that are manipulated by human hands. This seemed to get closer to the organic and directly-human heart of the project’s message.

I arrived at an odd set of objects that human hands could make cool sounds with. I whittled these down to only two objects: a kalimba and a one liter water bottle. The kalimba, of course, was played somewhat normally, but the one liter water bottle was tied to a string and swung by a microphone dozens of times, clapped, crunched, and blown upon. From all of these samples came a fairly small and concentrated set of sounds for positive feedback, errors, and transitions. I altered the volume envelopes on most of these sounds to make them either more pronounced or less percussive, and then applied some equalization and compression to make them all fit together, especially when played together…this is a multi-user application, after all.

Since the video above shows the map being used but doesn’t feature the interface’s audio, here’s a compilation of the actual sounds for your listening pleasure.

Sound Design of the Microsoft Local Impact Map for Surface by noisejockey
[Røde NT1a mic into Sound Devices 702 recorder, Soundtrack Pro and Peak Pro for post-processing]

Tags: , , , , , , | No Comments »