In a previous essay, “Points of View,” I suggested an artistic medium is a kind of lens, with unique properties that filter the audience’s view of the content. Media differ. Each medium features different types of representations, and thus different points of view (POV). Video games, for example, render perspectives that draw players into the onscreen action, often through the eyes of avatars. This idiosyncratic POV brings us into the space of the game world behind the screen. In that space, we see three-dimensions—there is depth behind the flat window—but, we cannot hear three dimensions through that window. Hearing requires sound waves to propagate outwards from the screen through the space we inhabit (through the medium of air) to our ears. Audio speakers project an artificial world represented within the game sound track, or on any sound recording, into real world space. Despite this fundamental property of sound recording, more and more we find screens providing space for listening experiences. In this way, onscreen imagery and action is giving listeners a new perspective on music.
Reverberation off of hard surfaces, ceilings, and walls; the pings and pops ricocheting off glass; and innovations like sound absorptive cushion are all examples of how the listening environment dramatically impacts the sound projecting out of speakers. While filmmakers do not spend a lot of time checking their work on iPads and laptops, sound engineers and record producers listen everywhere, on multiple speakers and on headphones. Projected large or small—in glaring sunlight or a darkened room—the shapes, proportions, and even colors (within a degree of variation) on a screen remain unchanged. Music is inexorably connected to space in which it is realized. Producers and engineers listen in different rooms and in the car revealing the effects of different listening spaces on their mixes, for example, the softening of a lead vocal’s bite or minor parts that unexpectedly come to the fore.
It would be wrong to think of space as a problem to overcome in music production. On recordings, space is as much a part of the music as pitch and rhythm. By reproducing performances that happened elsewhere in our homes and by creating virtual sonic environments with recording techniques like stereo and surround sound and signal processing, artists have been toying with our sense of space since the earliest days of sound recording.. Through the recording medium, artists are able to represent acoustic phenomena that are physically impossible to replicate in the real world: Loud instruments that can play soft instruments without masking them; one acoustic space superimposed on another, etc. These affectations are only possible with technological interventions, which are the source of new POVs, and provide us with ways to perceive music.
Screens, too, are a technological intervention shaping sonic space. They have become an ever-present component of sound reproduction. We listen to songs on YouTube. We play music apps. All our interfaces have screens. The problem is when listening to music in front of a screen (versus watching a screen and hearing music that accompanies the visuals) space collapses. The listeners’ attention is drawn into the flat surface despite the fact that the sound being realized in the air around them has depth and volume.
To redirect the listeners’ focus, some artists are experimenting with a new kind of depth—the space between the listener and the screen—one in which interaction is supported. Notable examples include projects featuring the music of Danger Mouse, Daniele Luppi and Norah Jones; Arcade Fire; and Björk have all found ways to bring space back into music listening, even though the listener is pinned to a screen throughout these experiences.
3 Dreams of Black
The interactive film “3 Dreams of Black” by video director Chris Milk showcases the music of Danger Mouse, Daniele Luppi, and Norah Jones. Though interactive, it very effectively evokes a mood of alienation and isolation, reoccurring themes in rock music. At the same time, it provides outlets for social connection, which is another facet of rock music’s appeal. In the film, solitary fans connect through the music (and artists) and through a desolate, dream-like wilderness.
After clicking “enter,” the first view of the world behind the screen is in the first-person perspective, but the imagery has a more cinematic than game-like quality. Occasionally, it pops out of the avatar’s perspective. (The essay “Points of View,” discussed how the film, “Coriolanus” popped in to the avatar’s perspective.) “3 Dreams of Black” is set in a post-apocalyptic someplace. It opens with a sketchy back-story, before opening up and letting the listener loose to wander in the fanciful and curious animated world. Listeners look around using the mouse. Elaborate patterns flower under the tracks of the movement. Minimal as it is, the interaction is lovely. Often in film scores, the music provides sub-context or a multi-faceted perspective on the onscreen drama. Here the flower motif set against the background of decay and moody music seems to function similarly.
Nothing the listener does controls the music. It is still a linear recording in the most conventional sense. As the song builds, the listener is spilled into an open space that offers greater opportunities for exploration (and to hear the song again). Flight motifs begin to dominate. The entire experience builds on a float-y, out-of-control sensation that is woven into various aspects of the scenery and the interaction. At the end, listeners can add objects to the world—customizing their “dreams” or leaving mementos; or they journey through others’ dreams, which ephemerally links listeners to other dreamers. “3 Dreams of Black” is not a game where play and interaction, as well as players’ ability to impact the game state, is essential for engagement.
“3 Dreams of Black” is not a music video, and it is not a film with a score. Figuring what can be controlled in the world, at times, diverts attention away from the gorgeously produced music. It is interesting that Milk et al. chose height, sky, and flight motifs. The illusion of height is notoriously hard to create using conventional stereo and surround sound recording techniques, and in this case, the visuals provided a sonic space with height that no conventional sound recording can match.
There is a "tech" page on the “3 Dreams of Black” site that offers demos of the technology, and pointers to related open standard development tools, as well as, the Google Chrome Experiments site, which features similarly adventurous projects. They also offer the source code for “3 Dreams of Black”, inviting new perspectives and exploration on several levels.
Arcade Fire’s “Wilderness Downtown,” also by Chris Milk and also a Google Experiment developed in HTML5, investigates personal space, geographic location, and the topology of the screen itself. Before launching the video, listeners are prompted to type in “the address of the home where you grew up.” Although anonymous, it is quite an intimate piece of information to be asked. Right at the start of the experience the listener is mentally transported to a(nother) location (and time).
The application incorporates images from Google maps into the video. The listener knows this wilderness. The interaction forces the song into the listeners’ personal history. Elegantly, powerfully and provocatively, if not subtly, the song invades personal space. The song literally occupies the listener’s real past.
“Wilderness Downtown” occupies real space in other ways too. Rather than cutting from scene to scene like a film or music video, different windows pop up across the screen. They layer on top of each other, creating both conceptual and physical depth. The entrances and exits of windows are (loosely) synchronized with the music, complementing the movement of musical parts from background to foreground.
Like “3 Dreams of Black,” bird motifs appear periodically to guide the listeners’ attention through the virtual; and again, impossible perspectives of height are juxtaposed against the music, this time rendered using Google satellite images. Rarely, in our listening lives do we listen from above; nor do we really in “Wilderness Downtown.” The visuals provide the allusion, a perspective on song’s meaning. But the mix, the acoustics in the sound recording offer no illusion of height, distance, or perspective. Instead, the music draws the listener in.
Arcade Fire has produced other, more app-like, interactive videos, as well. “Sprawl II (Mountians Beyond Mountians)” and “Neon Bible” by Vincent Morriset feature interactions tightly coupled to the music. These works seem to riff on single interaction motif rather than create a virtual world; but riffing is also a very musical idea.
“Biophillia,” the most recent album by the every ambitious Björk, is comprised of apps. The musical material references various nature themes and environmental processes. The organic forms featured in the visual motifs are evocative of biological and geological systems or entities. The music and the visuals play with scale, conjuring at times faint echoes of the influential Eames piece “Powers of 10” (1977). Björk’s virtual landscapes not only have their own geometry, they are filled with geometric shapes—literally polygons in virtual space that move in counterpoint with or frame the music. Space motifs, including outer space motifs, are prevalent throughout the album in the music and the visuals. Whether Björk is intuitively or consciously creating this space between the screen and the listener we cannot know.
Whereas traditionally recorded music forced artists to consider how various sonic elements compete for the listeners’ attention, now artists must consider how interaction, geometry, and the spatial relations represented on screens immerse listeners is a sonic space or draw them out of the music and into the visuals. How do the visuals impact what is heard in real, acoustic environments? This is something few sound engineers, record producers or recording artists have considered to date.
The prevalence of earbuds, computer speakers and screens have collapsed our listening spaces; but has also inspired artists to push out new directions. Interactivity and synchronized visual imagery have added depth to sound and new dimensions to music.