Why Include Sound in Digital Art?

Art departments in colleges and universities have traditionally divided their curriculum between art history and studio art, the studio art portion being devoted to drawing, painting, sculpture, printmaking, photography, and other visual media. Since the 1970s, however, as the computer has offered an increasingly attractive toolkit for artists, digital art has become another important component of many art programs.

It’s not surprising that digital art courses and programs – growing out of departments with a visual emphasis – might first focus on digital picture-making – digital photography, photographic processing, vector graphics, and paint programs. From these beginnings, the curriculum might go on to 2D animation and digital movie-making. Sound, however, is generally not a focus. Sound seems to be a more technical matter, the province of the engineer rather than the artist. Thus, sound often takes a backseat to the visual in digital art programs.

There are advantages, however, to introducing sound into digital art, even at early stages in the curriculum. Like image-making, sound can be studied historically and conceptually, and analogies between sound and visual art help to reinforce learning in these areas. In an historical approach, students can learn about the emergence of sound as one more technologically-based element of art, greeted with the same artistic skepticism that greeted the advent of photography. This can lead students to question the artistic principles guiding technologically-based art and the ways in which a new medium develops its own vocabulary and aesthetics. Considering these issues in more than one context – the visual and the aural – reinforces the students’ understanding and analytical ability.

The creative spaces in which digital pictures and sounds are edited also have parallels. For example, the difference between bitmap imaging and vector graphics is analogous to the difference between sampled digital audio and MIDI. Furthermore, the software interfaces for photographic processing, vector graphic drawing programs, and sound editing use similar terminology and metaphors. In imaging software, channels are color components that are mixed together in one pixel (e.g., RGB), while in audio software, channels refer to sound components that are mixed together at one moment in time (e.g., stereo). Layers in imaging software are analogous to tracks in audio software. Flattening an image’s layers is analogous to mixing down an audio file’s tracks. These analogies, again, reinforce student learning.

Another advantage of including sound early in the digital art curriculum is that it introduces another dimension into the students’ creative space – the dimension of time. While paintings, photographs, and their digital equivalents are static, sound is dynamic. Introducing the temporal dimension by means of sound prepares students for 4D design in digital video and animation, where timelines are a fundamental component in the software interfaces.

It isn’t difficult to be convinced of the value of sound in digital art courses, but sometimes the bar to including sound is simply that the faculty in a given art department have always worked in the visual arts. This is where their interests and talents lie, and they don’t relish the thought of retooling their labs and assignments to incorporate another medium. But retooling may not be as hard as it first appears, as we’ll try to show in what follows.

What Conceptual and Historical Background Is Relevant?

One way to approach sound, conceptually, in a digital art course is to view it from two perspectives: “sound in art” and “sound as art.” Viewing sound from these perspectives, possibly in an historical overview, can motivate students for experimentation in their own work.

“Sound in art” refers to sound as one component of an artistic piece, like sound in a movie, in a theatre production, or combined with still pictures used to illustrate poem or short story. Sound made a major appearance in art when it was first introduced into movies. Just when silent cinema had gained some respectability as an art form, sound arrived and changed all the rules. The reaction to any new kind of technologically-based art is basically the same – skepticism about its artistic legitimacy and a protectiveness of the uniquely human ability to create true art. With the invention of the camera, objections were made to photography-as-art, on the one hand because of the camera’s mindless literalism and on the other hand because of its potential for serendipitous success if a person simply takes enough pictures. Sound in the movies raised similar objections. The first uses of sound in movies were fairly literal and predictable. Pioneering sound artists devoted much of their attention to dialog and the technical challenge of synchronizing lips with voice. These new movies were, after all, called “the talkies.” The next obvious thing was to create realistic sound effects. An image of a clock called for a “tick tock,” and clapping hands called for the sound of applause.1

It was quickly realized that sound had much more potential than that. An interesting discussion of sound’s expressiveness appeared in a 1929 article by René Clair, where he described a scene from the movie Broadway Melody. In the scene described, sound clarifies meaning, freeing the images to maintain their artistic unity.

…we hear the noise of a door being slammed and a car driving off while we are shown Bessie Love’s anguished face watching from a window the departure which we do not see. This short scene in which the whole effect is concentrated on the actress’s face, and which the silent cinema would have had to break up in several visual fragments, owes its excellence to the ‘unity of place’ achieved through sound. (2)

In another example, René Clair describes how sound alone is used for meaning, not requiring an image. “In another scene we see Bessie Love long thoughtful and sad; we feel that she is on the verge of tears; but her face disappears in the shadow of a fade-out, and from the screen, now black, emerges a single sob.” (3) When sound is used “in” art, it interacts with other elements in a larger creative context. Sound extends space, time, and significance, going beyond the viewed image to places not visible. Arising from some cause, sound has a history and thereby can move the listener to the location and moment of its origination. Sound can be attached to, independent of, or in tension with images. Sound both clarifies and clouds meaning, depending on how it is used by the artist. Viewing and analyzing examples of sound in art can prepare the students for more meaningful creative explorations in their own work.

“Sound as art” emerged in the middle of the 20th century, when the artistic possibilities of sound broke out of the bounds of conventional music. Initially, sound technology was aimed primarily at capturing real-world, intentional sounds like talking, singing, and instrumental performances – and capturing them as faithfully as possible, “fidelity” being the desired quality. But the more the technology evolved, the more sound artists and engineers wanted to play and experiment. Electronic devices made it possible to filter, warp, reverberate, and otherwise alter sound as it was played. Magnetic tape became a new medium for recording, opening the possibility of cutting the tape and re-splicing it in all kinds of ingenious ways. This led to a new genre of called “electro-acoustic music.” (4) In France in 1949, Pierre Schaeffer coined the term “musique concrète,” a genre in which all sounds, not just harmonies and notes, were considered to be legitimate building blocks of music. Around the same time, John Cage spoke of “found” sounds, the sounds that exist in our environment. Cage encouraged the listener to pay attention to what they thought was silence, where sounds happen apart from the listener’s expectation and apart from a composer’s control. In place of the term “music,” Cage proposed the term “organized sound,” finding raw material not only in mathematically- related frequencies and regular rhythms, but also in what we might previously have considered mere noise. (5) The international Fluxus movement continued on this path, working with whatever sound material was available, putting the sound fragments together in a free-form and often minimalist style. (6) This break away from conventional harmonic music and interest in technological manipulation of sound paved the way for today’s hip hop, electronica, and digitally produced techno-music.

The legitimizing of sound as art was related to the Conceptual Art movement and a break away from conventional museum exhibits. Sound as art demands its own characteristic space because the environment itself is part of the creation. Art “installations” replaced museum exhibits. Art was experienced rather than viewed. N. B. Aldrich in his article “What Is Sound Art?” describes the experience of interacting with this kind of art.

The art installation…is an environment. It is a place where the audience is participating, or, rather, completing the art work through experiencing the environment. It is also the opportunity for an audience to engage work in their own time, the time they spend walking through the space, rather than be presented with a form that can be engaged only in its own time. The installation…invites the listener to create his or her own spatial relationship to the piece by moving through it and creating and individualized path or sequence. (7)

Considering both “sound as art” and “sound in art” requires students to deepen their listening skills and sensitivity to sound. Sound has a strong affective component in its ability to trigger memories, images, and emotions. Thus, part of the conceptual introduction to sound can include listening exercises that sensitize students to the origins, reverberations, associations, ambiguities, and significance of sounds, including those sounds that we have grown to ignore.

As is always the case in teaching, there is more to cover than time to cover it, so choices must be made. It’s possible to teach a course that is entirely historical and conceptual. On the other hand, the course could begin with concepts and historical perspective and continue with creative projects that reflect earlier discussions and analysis. The projects could challenge students to create “sound as art,” or they could ask students to integrate sound in a multimedia piece. In any case, the conceptual background can motivate the students to more thoughtful work, moving the focus from technology to meaning.

What Tools Are Needed?

But let’s get practical for the moment. What tools are needed for students to do hands-on work with sound? Getting ready to “do sound” – that is, getting the right computers, microphones, speakers, and software – may sound daunting at first, but it’s relatively easy. If the emphasis in a course is on concepts, design, and art and not on professional- quality production, an inexpensive sound system is all that is needed.

Sound has benefitted greatly from the digital video revolution. Ten years ago, finding a computer adequately equipped for digital sound processing was much more difficult than it is today. Now most computers are shipped with a digital video application, and this implies that they also can handle sound. Many computers have built-in microphones and speakers, making a basic laptop or desktop computer sufficient for simple sound processing and experimentation. For less than $500 (as of the writing of this paper), a system can be upgraded with a better microphone, external sound interface, MIDI keyboard, and headphones, as shown in Table 1.

Digital audio software is also readily accessible and fairly inexpensive if the basic level suffices. Basic sound processing software is often bundled with a new computer. Garage Band, an easy-to-learn music creation program – comes with every Macintosh computer. The next step up for the Mac would be Logic Express, which currently can be purchased at an academic rate of about $100. Cakewalk Music Creator for Windows costs only about $40. Garage Band, Logic, and Music Creator allow for audio and MIDI track along with loop-based composition. Garage Band and Logic are supplemented with large sample and loop libraries. Free audio software is also available. For example, Audacity – an audio recording/editing program compatible with Windows, Mac, or Linux operating systems – can be downloaded free from the web. More powerful, specialized software (e.g., Adobe Audition, Logic Pro, Pro Tools, Cakewalk Sonar, or Reason) is more expensive (in the hundreds), but generally a 30-day trial version can be downloaded, and it may be possible to schedule the assignments to fit in that time window. Current digital audio/MIDI software is listed in Table 2.

With more sophisticated tools becoming available, the minimum knowledge and skill set required to do digital sound has shrunk. The software is user-friendly, hiding the technical details or offering choices of instruments and audio features in convenient user interfaces. Thus, incorporating sound into digital art courses doesn’t require great technical expertise, nor does it rely on musical knowledge and ability. With the extensive loop libraries, even non-musicians can experiment with musical composition. The emphasis can be on the creation of sound as creative expression, whether it is associated with visual objects or exists independent of visual reference.

The point is that the hardware and software needed for dealing with sound in digital art courses are readily available and not prohibitively expensive. Needs will vary depending on whether the course requires a fully-equipped lab or works under the assumption that students have their own computers. Needs will also vary according to the types of assignments made. Many simple but meaningful assignments can be done using just Audacity, which is free.

What Technical Knowledge Is Needed?

Because of digital art’s technological underpinnings, digital art courses have to strike a balance between the technical, conceptual, and practical aspects of the subject. This continues to be true when sound is introduced. The amount of teaching time devoted to the technical and scientific aspects of sound depends, of course, on the goals of the course. Table 3 lists the fundamental concepts related to digital sound processing.

If an entire course is devoted to sound, then these topics can be covered in a systematic way. If sound is integrated into a digital art course along with imaging, then these topics can be covered on a need-to-know basis. For example, when students start a digital audio recording, they need to choose a sampling rate and bit depth for the file. To give a rationale for the choices, it’s necessary to explain (1) how sound can be represented as a wave with multiple frequency components (2) how sound is digitized through a process of sampling and quantization (3) why sound has to be sampled at a rate that is at least twice the frequency of the highest frequency component (4) how a low bit depth leads to quantization error. Later, as the students learn to edit sound – for example, adjusting the difference between loud and soft portions of a piece – it may be necessary to introduce the concept of dynamic range and its relationship to bit depth.

Textbooks that link the science and art of digital media exist, including interactive online tutorials that the students can work through in their own time. (8) Requiring students to read and practice with this material can free classroom time for other discussions and activities.

Ideas for Assignments

The assignment ideas below were collected from this paper’s authors and from participants in two workshops hosted by the authors in the summer of 2008. (9) The descriptions of the assignments have been abbreviated for this paper.

Contributed by Jason Romney, University of North

Project 1: Pick a children’s book and create a sound design for each page of the book. Make a music track using loops and add sound effects for each page. Scan the pages of the book and put them in as PowerPoint slides and then add your sound tracks to each slide in PowerPoint. You can record someone speaking the words or let the audience read it to themselves. In either case, each slide can have an underscore sound and a single sound effect that can be triggered by clicking objects in the slide. This project allows the introduction of the time domain without requiring the student to learn animation.

Project 2: Tell a story entirely in sound and music but without words, approximately two to five minutes long. Use digital audio, creating your project in a multi-track editing environment. MIDI is optional. Apply special as appropriate effects to create your desired atmosphere and emotions. The sounds should provide a narrative structure. Play your piece without explaining the intent of your narrative, and have the listeners tell you what story they hear.

Contributed by Ken Baldauf, Florida State University

This assignment is intended to illustrate the difference, both intellectually and through object listening, between music that has been sampled at different rates using 8 vs. 16 bit samples values, mono vs. stereo samples, and compression vs. no compression. Begin by ripping one of your favorite songs from CD using the following format: wav, 16 bit, stereo, at 44.1 kHz or higher. Encode seven more copies of your song in the following formats: 8 bit mono, 6 kHz wav; 8 bit stereo 44.1 kHz wav; 16 bit mono 6 kHz wav; 16 bit stereo 6 kHz wav; 16 bit stereo 24 kH wav; 320 kb/s mp3, and 160 kb/s mp3. Without looking at file details, use what you’ve learned about digital sound to guess the file size and sound quality of each of these versions (ranking them from best to worst). Check the file sizes. Listen to each sound file and make notes on differences you perceive in the quality. Compare this to your expectations. (This could be done in any basic audio editing program that has the capability of changing sampling rate and bit depth.)

Contributed by Brian Evans, University of Alabama

This assignment introduces sound and sound design techniques of musique concrète (which we’ll call “sound collage” in this assignment). This assignment will require that you think about time-based media and explore temporal structure. Create a stereo sound file composed of at least three different sound sources (and not too many more). The finished file should last exactly 90 seconds. Also hand in (1) a score (graphical and notated) that outlines the composition of the file. (See Notations by John Cage and other examples to be cited by the instructor.) Incorporate signal processing techniques such as EQ and pitch shifting. Explore at least three standard “classical tape techniques.” Give the file a coherent formal structure. Use multi-track recording. (This could be done using SoundTrack, Audacity, and/or Garage Band. Signal processing packages like TimeToy could also be used.)

Contributed by Joe Pino, Carnegie-Mellon University

The object of this assignment is to explore sound textures and manipulate timbres. Using the sound source files provided, you are to edit, mix, and manipulate as many different sounds as you can produce in three hours. Then, using these files, sequence the sounds into a three minute soundscape that tells a story in the sense that there is a change from one mood to another or some sequence of “cause and effect.” You can use functions to change pitch, duration, or gain; convolve; modulate; reverse; fade in/out; convert sample rate; or phase vocode. Experiment with ways of combining the original sound with one or more of the additional sounds to create new sounds that complement the original mental image, augment it, or alter it to something new. Focus on creating sounds with complex and interesting texture and color. Don’t be concerned about what a sound is; only how it is. Do not spend more than five hours on this project, three to create the source material and two to mix a final soundscore. (This could be done with basic audio editing software.)

Contributed by Conrad Gleber, La Salle University

Select a poem with at least 100 words. Select 20 to 30 words in the poem that are descriptive of action (windy, splash, cracking, crunch, etc.). List the words and give each of the selected words a sound (no music allowed). The result is a group of sounds that are placed into an audio track. Associate each sound with a picture on the video track. Now design and manipulate the audio and pictures to create a combination of images and sound that is a more cohesive, albeit abstract, work. (This could be done in Director, Flash, Final Cut, or Premiere.)

Contributed by Roy Carter, Wake Forest University

Week 1 (Design) Choose an experience (based on a real or imagined interaction) that produces a change in mood, idea, attitude, or physical orientation. This experience will serve as the inspiration for a collection of images, sounds, motions, and interaction techniques that you will create with authoring software. Create a lo-fidelity prototype for the experience. Allow for experimental user interaction that is intended to expand the user’s understanding of the moment. Decide the elements that you will have control over and those that the user can control. Week 2 (Building Components) Experiment by building a palette of sounds and imagery. Limit our selections by evaluating the potential of each for communication. The sounds may not be representative of objects within your images, and the images may not be directly representative of objects within the experience. Open your mind to the abstractions. Your focus experience may be the moments surrounding the situation that you are referencing and not the situation itself. Layer the sound in the experience for richness. Week 3 (Laying Out the Design) Lay it all out in your authoring environment. Be sensitive to relationships that are naturally happening in your imagery and audio. Don’t be heavy-handed with your message. Let it develop out of your process. (This could be done in Director, Flash, Final Cut, or Premiere.)

Contributed by Jennifer Burg, Wake Forest University

This type of project could be done with any number of stories, but we’ll use William Faulkner’s “A Rose for Emily” as an example. This story tells of Emily Grierson, a proud Southern woman at the turn of the century who sees her best years pass her by as she ages into a lonely reclusive spinster. When Emily dies, it is discovered that she has kept a corpse locked in her bedroom and lying upon her bed, probably the Yankee construction worker who jilted her years ago. Read the story and discuss it with the class, including the story’s treatment of time and Emily’s desire for things not to change. Create one digital image to illustrate this story and express the theme that you think is central to it. To go along with this picture, create an audio piece that also illustrates this theme. As you create the audio piece, consider how sound can be used to extend the boundaries of a picture in both space and time. For example, if the picture is of the inside of Emily’s house, consider what is going on outside. You can use both recognizable and unrecognizable sounds, since the fact that a sound is not recognizable might also convey meaning. Use digital audio, creating your project in a multi-track editing environment. MIDI is optional. Create all your sound from original recordings or from the library of public domain sound effects made available to you. Do not take sounds from web sources. You can put our final production in PowerPoint. The PowerPoint presentation should begin with a title page, and the sound should be set to play automatically when the user goes to the image after the title page. (This could be done with basic audio editing and MIDI software such as Cakewalk Music Creator coupled with PowerPoint.)

Footnotes:

1. James Lastra, Sound Technology and the American Cinema. (New York: Columbia University Press, 2000).
2. René Clair, “The Art of Sound,” in Elizabeth Weis and John Belton (eds.), Film Sound: Theory and Practice (New York: Columbia University Press, 1985), 92-96.
3. Ibid.
4. Herbert A. Deutsch, Electroacoustic Music: The First Century (Van Nuys, CA: Alfred Publishing Company, 1993).
5. Leigh Landy, Understanding the Art of Sound Organization (Cambridge, MA: The MIT Press, 2007).
6. Hannah Higgins, Fluxus Experience (Berkeley, CA: University of California Press, 2002).
7. N. B. Aldrich, “What Is Sound?”, http://emfinstitute.emf.org/ articles/aldrich03/aldrich.html.( accessed March 3, 2009).
8. Three companion books that link the science and art of digital media are Jennifer Burg, The Science of Digital Media (Upper Saddle River, NY: Prentice-Hall, 2009), Y. L. Wong, Digital Media Primer (Upper Saddle River, NY: Prentice-Hall, 2009), and Y.L. Wong, Digital Media: Its Art and Science (Upper Saddle River, NY: Prentice-Hall, 2009). See http://prenhall. com/digitalmedia/.
9. See http://www.cs.wfu.edu/~burg/CPATH/WFU_Workshop/ index.html and http://www.cs.wfu.edu/~burg/CCLI/Workshop1/ for workshop proceedings.

Bibliography:

Aldrich, N. B. “What Is Sound?” http://emfinstitute.emf.org/ articles/aldrich03/aldrich.html.
Burg, Jennifer. The Science of Digital Media. Upper Saddle River, NY: Prentice-Hall, 2009.
Clair, René. “The Art of Sound.” In Elizabeth Weis and John Belton, eds. Film Sound: Theory and Practice. New York: Columbia University Press, 1985, pp. 92 – 96.
Deutsch, Herbert A. Electroacoustic Music: The First Century. Van Nuys, CA: Alfred Publishing Company, 1993.
Higgins, Hannah. Fluxus Experience. Berkeley, CA: University of California Press, 2002.
Landy, Leigh. Understanding the Art of Sound Organization. Cambridge, MA: The MIT Press, 2007.
Lastra, James. Sound Technology and the American Cinema. New York: Columbia University Press, 2000.
Wong, Yue-Ling. Digital Media: Its Art and Science. Upper Saddle River, NY: Prentice-Hall, 2009.
Wong, Yue-Ling. Digital Media Primer. Upper Saddle River, NY: Prentice-Hall, 2009.