Forum 3: The State of the Art
Prescriptions for the Present
Photo Credit for Featured Image: © Fischinger Trust
Editor’s Intro:
For this final forum, I invited short contributions from practitioners of audiovisual art working outside of the realm of cinema proper to offer some thoughts on the current state of the art, and what the future might hold for artists interested in achieving higher levels of integration between sound and image.
Fred Collopy begins by taking us back to the golden age of instrument invention that gave birth to the emerging form that has come to be known as Visual Music. He suggests that with the dawn of modern computer technologies in the 1970s, we have returned to an age where the invention of new audiovisual instruments is on people’s minds. He hopes that with many of the difficulties of centuries past now overcome by contemporary technology, we may yet find a new form of audiovisual art that can stand the test of time.
We then move to two essays which offer prescriptions for getting beyond the conventional trappings of audiovisual art, and perhaps move us towards the new form suggested by Collopy. Michael Betancourt begins by identifying conventions of sound-image relationships in Visual Music that have tended to be the reverse of what has arisen within the cinema: image is subservient to sound rather than sound being subservient to the image. In both extremes, conventions of tight synchronization on certain aspects of sound/image relationships have become the dominant formal strategy. He suggests alternatives that fall somewhere between these two poles. Then, Barry Spinello ventures into the realm of cognitive perception. He draws on the idea that our minds do NOT separate sensory information according to the five senses in order to suggest ways in which the practice of audiovisual creation might best resemble the lack of sensory boundaries that exist on the level of mental processing.
What these three essays ultimately have in common is a desire to see the increasingly artificial distinctions between “sound” and “image” broken down so that we might strive towards a level of integration that would finally yield an art form in which the treatment of the heard and the seen is inherently equal. There are glimpses of such an art form throughout the history of cinema and its parallel media, yet many would argue that the ultimate goal of perfect equilibrium is still a long way off. Finally, these essays point to what might best be called “designing art for audiovisual integration.” For those interested in such a goal, this forum offers suitable closing thoughts for this special issue of Offscreen.
Thanks for reading. -RJ
__________
Visual Music as a Performing Art
By Fred Collopy
With the invention of the ocular harpsichord in 1725, Louis-Bertrand Castel attempted to integrate sound and image in real-time. Almost three hundred years later, as the paths of inventors have crossed with painters, musicians, and filmmakers, we are on the cusp of realizing the potential for an audiovisual performing art on par with the other great mediums of history.
Castel believed that there was an analogy between sound and light that could serve as the basis for a visual art that was as powerful in its emotional appeal as music. Though sound and light share certain characteristics, there are also important differences between them. Tones merge while colors remain separate, and tones are fleeting while colors persist. Since tones could not be made persistent, Castel proposed to make colors transient and use them in combinations to create visual harmonies.
While inventors worried about making colors move in time, painters were devising bases for arranging colors in appealing combinations. Some did so in the specific context of color music. Between 1912 and 1914 Leopold Survage painted over 200 watercolors intended to form the basis for an abstract film, Colored Rhythm. Both its colors and the movements implied by its changing forms are stunning, especially given that it anticipated both color stock and abstract animation by years.
With the addition of form, Survage changed color music into visual music. Devising a language for their movement was immediately important. “An immobile abstract form does not say much…It is only when it sets in motion, when it is transformed and meets other forms, that it becomes capable of evoking a feeling” (quoted in Graves 1951:412). Soon after, Thomas Wilfred’s invention, the Clavilux, also incorporated form and motion. “Form, color and motion are the three basic factors in lumia – as in all visual experience – and form and motion are the two most important,” Wilfred wrote (Wilfred 1947:252).
The 20th century continued to see the invention of visual instruments, but this era belonged to the filmmakers. Viking Eggeling and Hans Richter were scroll painters who turned to film in order to make their forms move. Following their 1919 debuts, dozens of filmmakers created abstract films that addressed such visual problems as counterpoint, continuity, tension-release, figure-ground, implied causality, and changing perspective. But film is fixed, each showing of a film very like every other, while the performance of live music is not.
Though the goal of color music’s inventors was to create an art as powerful as music, few composers committed themselves to any significant degree. Telemann was supportive of Castel’s ocular harpsichord, expressing certainty that its play of colors would please. Schönberg, Scriabin, Baranoff-Rossiné, Stokowski and Varèse experimented in modest ways with its potential. But musicians’ attitudes are now changing as they become increasingly interested in controlling the visual element of their performances.
With computers we are again in an age of the instrument inventor. What was difficult in the 18th century is no longer so. Moreover, we now have three centuries of thinking and experimentation to guide us. This art, while not entirely imitating painting or film, music or dance, draws on each of these. Audiences are watching, and musicians are playing along, as we create abstract visuals that move. Perhaps technology has now caught up with early ideals, and we will finally find a new art form in audiovisual performance on par with the time-tested appeal of music alone.
Sources Cited:
Graves, Maitland. (1951). The Art of Color and Design. New York: McGraw-Hill Book Company.
Wilfred, Thomas. (1947, June). Light and the Artist. Journal of Aesthetics and Art Criticism, V, 247-255.
__________
Synchronous Form in Visual Music
By Michael Betancourt
Throughout the history of visual music, both on film and in other media, there has been a remarkable consistency in approaching the question of how to synchronize sound and image: structure is provided by a pre-existing piece of music, the form of which is systematically translated into the visual realm. Early in the sound film era, Sergei Eisenstein discussed how his montage aesthetic in the “Battle on the Ice” sequence in Alexander Nevski is intended to be a visual dramatization of the Prokofiev composition that forms the soundtrack (Eisenstein 1975). Eisenstien’s approach here is essentially no different than Oskar Fischinger’s animation of shapes and forms synchronized to music. In both cases the musical form is taken as given, which then determines the structure and arrangement of the visuals designed to accompany it.
In the realm of visual music, approaches to sound and image rarely move outside the established framework of synchronizing visuals to a predetermined score. Experiments in the direct and simultaneous translation of image and sound are rare, but they do exist. Take the work of the Whitney brothers, or Norman McLaren, or more recently by composer/CGI animator Dennis Miller; in the hands of these artists, sound and image are often technological representations of the same thing. These examples are exceptional precisely because they are atypical. However, the resulting works these experiments produce serve to “naturalize” the arbitrary nature of their construction, implying that the only appropriate relationship between sound and image is one where they are locked into an absolute, synchronous form. In works such as these, there can be no counterpoint between sound and image. Technologically speaking, whether recorded on film/video, or “live” in an interactive environment as with contemporary work by Golan Levin or Leo Villareal, works such as these present approaches to sound/image interaction akin to lip-sync in dramatic film. The music and the image correspond to each other upon precise points of conventionalized synchronization.
The influence of Fischinger’s work with tight synchronization cannot be underestimated. Close synchronization is the most common variety of visual music partly because it is the most commercial. William Moritz has noted in his discussion of Fischinger’s films that he used “tight synchronization partly because of his commercial ties with record advertising and partly because he found that audiences would more easily accept abstract visual art if it were linked to recognizable music (abstract auditory art) they already approved of” (Moritz 1996:228). The implication of this comment is clear: abstract visuals present particular difficulties for audiences. So in order to ensure the commercial viability of his films, Fischinger was forced to adopt an approach dependent upon the use of familiar auditory material as the foundation for his visual creations. And many have followed his lead in creating works of visual music based on the subservience of image to sound.
Furthermore, it should be observed that music is rarely heard as a collection of disparate pieces, but is instead understood as an unfolding over time. The linkage between images and sounds, where one image is changed for another following the music in a rhythmic montage, results in a visual music that is extremely distant from what is heard. In emphasizing the rhythmic aspects of music, all the aspects of harmony, melody, and development over time visually disappear. While the rhythm-based approach may be very common, it ultimately offers the fewest aesthetic potentials for visual music.
Alternatives are possible. Image and sound can be synchronized in any number of ways other than the direct linkage of note to visual motif on points of rhythm. The phrase and measure can be treated as distinct entities analogous to shorter blocks of sound upon which synchronization is conventionally based; separation of image components can be based on the components of musical instrumentation; and the relative dynamics of the music can serve as guidelines for dynamic equivalencies within the image. John Whitney’s ideas about “visual harmony” illustrate another approach to creating visual music, one where the structural design of both image and sound proceed from the same basis; they do not directly correspond in the manner of tight synchronization, but they are founded upon the same principles (Alves 2005). Each of these potentials would qualify as “visual music” but would not necessarily follow the specific parameters of tight synchronization or subservience of one realm to the other. Instead, the result would be to introduce the potential for sound and image to exist in a relationship of counterpoint that moves beyond the matching of sound and image on points of rhythm, and beyond the need to place one sense above the other in the hierarchy of compositional organization.
Sources Cited:
Eisenstein, Sergei. (1975). The Film Sense (Jay Leyda, Trans.). New York: Harvest/HBJ, 174-216.
Moritz, William. (1996). Visual Music and Film-as-an-Art Before 1950. On the Edge of America: California Modernist Art, 1900-1950, Paul J. Karlstrom, ed. Berkeley: University of California Press, 228.
Alves, Bill. (2005, Winter). “Digital Harmony of Sound and Light” in Leonardo Computer Music Journal, 29, 4, 45-54.
__________
On Sound and Image as a Single Entity
By Barry Spinello
The brain’s mechanical apparatuses for seeing and hearing are apparently separate and fixed in place. But there is a huge interconnectivity possible between these two areas and it is this interconnectivity that should be of interest to artists making audiovisual work. Yet as we know, the breadth of possible interaction between sound and image has not lived up to its potential. Exploring this interconnectivity through audio-visual composition has fallen into a dependence upon very particular conventions of sound/image synchronization. The potential for this problem to arise was identified in the earliest days of the sound film, and the alternatives have been used only sparingly in the decades that followed. I suggest that positioning the making of sound and the making of image as aspects of the same process yields valuable insight into compositional practices for the creation of audio-visual art which, if more widely adopted, could result in a broader range of audiovisual exploration than exists today. With this in mind, I would like to address the following question: how might sound and image be conceived of, and composed, as a single entity?
Imagine the following scenario: the art object, be it a painting or composed piece of music, is not “the thing”. The object is but a residue of “the thing”. The real “thing” is the interneural connectivity within the artist that caused the object to come into existence. So, when a spectator sees the object that the artist made, there is the possibility that a roughly equivalent set of interneural connections is built within the spectator that corresponds to the set of interneural connections in the painter during the act of creation. Thus the potential exists for the mind of the artist to become one with the mind of the perceiver through the art object itself. This suggestion brutally simplifies the process of experiencing a work of art. Yet the communication of two minds through the experience of art is fundamental to art’s purpose. We get at each other’s minds through the process of externalization that yields concrete artifacts within the world.
But how do these concrete artifacts get made?
Let’s imagine another scenario: the composer creates a piece of music through the motor act of playing a note, then playing another note, followed by a third note, etc. Similarly, a painting is composed by a similar process; one line follows another and so on. In either of these scenarios, a fraction of a second may occur between the discreet notes or markings being used as compositional elements. In that fraction of a second a myriad of interneural connections can come into play referencing the artist’s history, training, feeling, personality, what was had for breakfast, etc. But one thing is clear – if a second note is to follow the first, the competing interneural activities must resolve into a single dominate strain, which releases as the motor act of the next note.
We may be tempted to believe that if the artist is a musician, he or she is working primarily within the gestalt of hearing. That is to say, by training, custom and practice, the interneural connections within the artist circle primarily around the act of hearing. And we may hold similar beliefs regarding the painter’s emphasis on the act of seeing. Such beliefs are founded upon the notion that we have separate worlds within us that are defined by our external sensory receptors, and that these worlds are fundamentally isolated along the lines of seeing, hearing, etc. So, within us we have these two seemingly separate worlds: the world of seeing, the world of hearing; the functionality of the eye, and that of the ear.
One World out of Many
Music and paintings that are created separately can be played simultaneously, and when that happens they reinforce each other. Our powerful drive to seek associations between sound and image forces a certain measure of audiovisual synchronization when these elements are presented together. Countless movies have been strengthened in emotional content simply by playing a Bach track alongside the pictures. Inversely, music videos start with a coherent piece of music, written as music, to which pictures are then added. Fresh results can happen even when picture and sound are randomly combined. But the synchronization of two different forms, created separately and with different tools, leaves painting within the gestalt of seeing, and music within the gestalt of hearing. This separation at the point of creation begs the point of a fundamental integration.
Perhaps what is needed to forge such an integration is a compositional practice based on the idea that our minds do NOT separate sensory information according to the five senses. Perhaps the potential integration between hearing and seeing is something inherent to the way we make sense of the world rather than something that needs to be forced. If we approach artistic disciplines in the same way, recognizing inherent similarities rather than emphasizing differences, we may move towards an art practice that understands the “audiovisual” as a single entity.
Towards a conclusion:
Music and painting can’t be treated as the same thing if the auditory and visual portions are created on different instruments. Similarly, the melding of sound and picture must come at the level of conception within the creator, not as an afterthought.
So here are some suggestions about how to avoid the trappings of sensory separation in audiovisual art:
1. Use a single program (like After Effects) in which both sound and picture can be manipulated or arranged using the same method (moving pixels) in the same time-line.
2. Start with the smallest jot of sound and the simplest impulse of picture. Move and arrange the pixels of sound and the pixels of picture in and around each other, frame by frame, according to your own personal aesthetic – a new audiovisual aesthetic that you must build from the ground-up within yourself.
3. Strive to proceed with sound and picture together, not relying even for a small passage on your wonderful capacity to make music, or your long history of drawing lines into picture. Rather, jump the track of the visual gestalt and steer off the custom of the audible gestalt to form a new interneural pathway. A pathway that integrates seeing and hearing at the level of creation… A pathway that, once achieved, can be absorbed by viewers so that each successive step towards the new audio-visual art form is solid, differentiated and additive.
*This essay is an abbreviated adaptation of an earlier version, available in full here: http://www.auzgnosis.com/pgs/intoactn.htm.
PDF Downloads
- Visual Music as Performing Art by Fred Collopy
- Synchronous Form In Visual Music by Michael Betancourt
- On Sound and Image as a Single Entity by Barry Spinello
