Color data is translated to audio. To hear the additive synthesis most distinctly, choose the the color palette. Drag horizontally to hear changes in the intensity of a single frequency. Drag vertically to hear the effects of multiple colors together. Though it is a single waveform, you can distinctly hear multiple tones moving in opposite directions as the colors transition.

Sonified Trace by Jack Stockholm

Exploring visual data with sound allows perception of elements that the eye cannot see. "Sonified Trace" engages the interface between digital photography and the human visual explorer by means of sonification of photographic color data. Ideally patterened using a touch pad mouse, the visual explorer drags a finger across the image and leaves a line tracing their path across the colors of the photo. This trace is then sonified and each pixel along the path is played back to the "performer" as sequential audio. The user not only hears the pixels under their trace, they also can see the result of a shakey mouse finger and hear where the pixels under their path may not be the ones they chose. Sonification is being used in a variey of perceptual analysis, including telescopic astral data and satellite imagery, to discover what other senses are not able to amalgamate.

The algoritm used here is rather simple and the tool crude but even this blunt instrument serves to engage the ear where the eye cannot see. Each pixel is represented by three colors: red, green and blue. The amount of red data in a pixel is translated to the frequency of a triangle waveform, with a low amount red equalling a low frequency wave and a high amount of red equalling a high frequency wave. Green data is translated into a square waveform and blue data is translated to a sine wave. The three are then combined to produce a single waveform that is played back. The ear can single out changes in the waveform landscape and can learn to discover elements to which the eye is blind.

This project allowed several new programmatic challenges utilizing the IDE Processing. First, I began developing a hierarchy of GUI classes that would allow me to engage the end-user through their manipulation of controls. The bottom-most class is CControl, a class from which all other visual classes are derived. CControl forces all child classes to implement a .show() function which draws itself on the screen. The next class, CModdableControl, is a class that inherits from CControl but adds abstract function .pointIn(). This function, required by all children, tests if an x,y coordinate is within the boundaries of the control. These two base classes allow an essential programmatic function, the ability to add all objects to an ArrayList and loop through that list to draw the controls and test for mouse manipulation. I then set about drawing visual representations for these controls.

The primary goal of this assignment was to create a GUI that references our cultural bias. The end-user's operation of this instrument would reveal the underlying predjudices in the tool. My personal goal in this assignment and where I remain grateful in the authorization to proceed is my exploration of sound and how the user adjusts their process to the results of the audio generated from their drawing actions. I implemented Krister's ESS sound library for Processing to generate the audio. I had the opportunity to explore audio stream objects with cached wave buffer data to which I sent generative audio synthesized from the image data. I synthesized all of my own wave data so that I could add them together rather than using the predefined Sine, Triangle and Square objects. The result achieves my goal of producing more significantly for the ear what can barely be seen by the eye.

Source code: Project_2

Built with Processing