THE ART OF INTERFACE

One’s experience of the world is mediated on numerous levels. Not only do we define objects and actions based on prior history and personal memory coalesced into a coherent communicable whole by language, but we are also influenced by the mechanisms that catalyze our very actions into experience itself. In fact, a preponderance of interactions we thoughtlessly engage in join together man and machine. Yet this cybernetic bond is not recognized as the dynamic employed places the user in a purely responsive position to the machine. As a result, the physical elements are dependant on the prompts and cues supplied by the machine existing in virtual space. Furthermore, the points of exchange, where information is passed between the virtual to the physical are constructed to negate any real dialogue and thus equality between the user and the source. Coupled with the growing impossibility to differentiate between the two raises serious concerns on the ability of the individual to exist as one freely chooses.

The Graphical User Interface (GUI) is an ideal example of how a mechanism, constructed to solidify the social fabric and institutions, mediates experience. GUI’s, while found in an array of ever-popular hand-held devices and in nearly every financial transaction, are most notably associated with personal computers. Their structure is based on cultural metaphors taken from a mixture of already established forms of media and the workplace. In fact the GUI took on a more encompassing role as media theorists Lev Manovich writes in his text The Language of New Media:

“In the 1990’s, as the Internet progressively grew in popularity, the role of the digital computer shifted from being a particular technology (a calculator, symbol processor, image manipulator, etc.) to a filter for all culture, a form through which all kinds of cultural and artistic production were mediated. As the window of a Web browser replaced cinema and television screen, the art gallery wall, library and book, all at once, the new situation manifested itself. All culture, past and present, came to be filtered through a computer, with its particular human-computer interface.”[1]

While the incorporation of metaphors and the ability to democratize culture has led to the widespread integration of the computer, the repercussions of implementing constructs that relegate the user to just that perpetuate the continuation of isolated centers of knowledge that are inaccessible to the remaining masses by eliminating innovative and frameless approaches to the bridge between physical and virtual realities.

The creation of open-ended dialogues between interface and user has been the constant inquiry in the ever-evolving investigation of David Rokeby; artist, engineer, and researcher. His body of work to date highlights the cybernetic exigencies while simultaneously challenging the archetypes of the graphical user interface. For Rokeby, the event that is interactivity must be responsive; that being, co-authored through an on-going open discourse of information and knowledge rather than defining understanding internally through a semiotic interpretation dictated by structures of power. To reach this goal, Rokeby’s systems are carefully constructed from the bottom up; emphasizing process in both their all encompassing feedback loops and the focused codification of responsivity in his systems specific to the multi-modalities of experience in space-time. Therefore, experience in an un-mediated world is discovered by the responsivity of interaction and the cumulative product of the discursive exchange of information and knowledge between the confines of reality and the vastness of virtuality. The interface cannot be defined by representations of the predominant power dynamic. It must be free to explore the inexplicable similarities of consciousness and cyberspace.

Very Nervous System (VNS) is David Rokeby’s most known interactive installation. When the viewer enters the physical space of the installation, VNS interprets the viewer’s motion into audible sound. The movements are detected by a video camera opposite the viewer. The image is then sent to a computer where the image is processed into a signal that is in turn sent to a synthesizer that outputs the signal into sound that is received by the viewer via speakers. VNS continuously responds to each and every movement received by the viewer and likewise the viewer responds to the sounds of VNS. The result is that the interface is no longer mediating the experience; rather, it is transparent as the viewer and system form an indexical function through a 1:1 relationship established by Rokeby’s careful coding of VNS’s language and the willingness of the viewer. One or the other does not dictate the communication, it is a discursive dialogue where information and knowledge of the experience are equally exchanged enhancing the perception of qualities and informing the user on the connectiveness that surrounds us.

Rokeby’s decision to place the interaction on such a large scale in physical space using the motion of the entire body was very deliberate. He wrote:

“I created the work for many reasons, but perhaps the most pervasive reason was a simple impulse towards contrariness. The computer as a medium is strongly biased. And so my impulse while using the computer was to work solidly against these biases. Because the computer is purely logical, the language of interaction should strive to be intuitive. Because the computer removes you from your body, the body should be strongly engaged. Because the computer’s activity takes place on the tiny playing fields of integrated circuits, the encounter with the computer should take place in human-scaled physical space. Because the computer is objective and disinterested, the experience should be intimate. The active ingredient of the work is its interface.”[2]

Rokeby not only adapted, through the language of the system, how the interface functions, but also how the user herself interacts. This was accomplished by equalizing the amount of information given by the user to that outputted by the system eradicating the incongruencies apparent in most common GUI’s. That being, the sparse clicks of the mouse and keyboard are never directly proportionate to the functions of the machine. By equalizing the playing field between user and machine in the interface of the installation, Rokeby establishes an experience where discourse can occur.

While Rokeby’s intentions for an open-ended continuous dialogue between user and interface is accomplished, the obvious transparency led many users to misinterpret and come to alternate conclusions. There are two main examples of how this schism between artistic intention and viewer interpretations occurred. The first is a trope that many interactive installations have fell victim to; that being, the viewer plays with the installation excluding the possibility for meta-narratives to arise. In limiting the experience solely to the surface level of cause and effect, one can argue that Rokeby’s motivation of wanting to highlight the cybernetic exigencies due precisely to the lack of recognition of the bond formed with interfaces in contemporary society justified. The users simply do not see past themselves negating the interface as they are minimalized by the predominant structure of user interfaces within society.

This presumed dominance of the viewer of the VNS leads to the second common negative interpretation, that being, it is not truly an interactive system. This arises from the system’s apparent failure of what is referred to as the test of interactivity. A simple process of verification that was in fact too simple for VNS. Rokeby writes of this in his essay “The Construction of Experience: Interface as Content”:

“They would enter the space, let the sounds created by their entrance fade to silence, and then make a gesture. The gesture was an experiment, a question to the space; “What sound will you make?”. The resulting sound was noted. Second and third gestures were made with the same motivation, and the same sound was produced. After the third repetition, the interactor decided that the system was indeed interactive, at which point they changed the way they held their body and made a gesture to the space, a sort of command: “Make that sound.” The command gesture was significantly different from the early “questioning” gestures particularly in terms of dynamics, and so the system responded with a different sound. I observed a couple of people going through this cycle several times before leaving in confusion. Their body had betrayed their motivation.”[3]

In ignoring the multi-modalities that are existent in interpersonal communication, the user is unable to reconcile the disparity in sounds for the assumingly similar motions. In fact, this ability to differentiate between symbolic and gestural qualities is what endows VNS with the ability to maintain a dialogue that is worthy of a discursive label. For if the installation were to merely respond equally to what Rokeby called question and command gestures, it would not be a discourse of knowledge but a responsive system that does nothing to increase and distribute knowledge; in this case, the experience of interaction with the space of the installation. The cumulative result of these interpreted misunderstandings had a definite impact on Rokeby’s future work.

While still employing transparent interfaces, Rokeby does not make them so glaringly apparent and pushing it farther, minimizes the individual’s role. In order to further explore and effectively convey his research on the roles of interfaces and their influence on users, these work creates experiences that demand the viewer to investigate and interrogate the events outside of themselves in favor of that which the interface connects; physical and virtual space. Two projects in particular demonstrate Rokeby’s evolution from Very Nervous System, Silicon Remembers Carbon and The Giver of Names. Furthermore, Rokeby calls attention to the impact that memory and language have on our perception to address the confusion that arose from the viewer misinterpreting or disregarding the complexity and met-narratives that reside with in his system.

While maintaining motion detection in Silicon Remembers Carbon, Rokeby does not make it the central focus as the immediate attention of the viewer falls upon the sculptural video installation pushing the interaction to the edges of perception. The video installation component consists of an overhead projection onto a bed of sand. In using sand as his screen, Rokeby is able to create an illusion of depth that heightens the convergence of the physical and virtual spaces. The projection itself consists of various sequences of water (in close proximity to bridges) mixed with pre-recorded and real-time streamed reflections deriving from either passer-bys while Rokeby initially recorded the water or those of gallery visitors where the installations was on display. The mix was dictated by a combination of viewer presence, interaction with the physical boundaries of the installation, and the movements within the gallery space by a computer informed by infrared sensors strategically position around the space, but out of view from the viewer.[4]

In contrast to Very Nervous System, it is quite conceivable that the viewer would not recognize the relationship between their actions and the systems (what they are seeing) much like most people ignore the cybernetic bond found in interactions with graphical user interfaces. Going further, Rokeby mirrors the movements of users with the pre-recorded reflections and at other times inserts the reflections of actual users. Rokeby himself noted the different approach he undertook in his “Lecture for “Info Art” at the Kwangju Biennale:

“First, I wanted to increase the probability of certain conjunctions of events in relation to the audience’s actions such as the apparent mirroring of live person by pre-recorded shadows in the image, but I didn’t want the audience to start performing to the image. Second, I did not want to overwhelm the other, non-technical interactions in the space. The work is full of ambiguous propositions that must play out in the mind of the audience: Is that my shadow? Should I step into the image? Have I violated the artwork? Is this real? Did I help create this?”[5]

Therefore the audience does not only have to discern a cause and effect relationship, but must allow for the meta-narratives to surface in order to understand the complexity of what they are experiencing. It is not just a dialogue that occurs, but also a conscience inquiry into how the body and its actions inform the system and how the nature of the system’s interface defines the perception of the individual during the experience.

The Giver of Names progresses the notions developed in Rokeby’s prior work to new levels by minimizing the user’s physical interaction. In the work, the viewer’s sole physical form of interaction is that of placing an object on a pedestal where a video camera captures an image of the object that is subsequently processed by the system ultimately outputting back to the viewer a phrase or a name so to speak. Erkki Huhtamo notes three descriptions give by the system to three relatively similar objects:

“the fiver of names for example described a small yellow rubber ducky as “Semicircles, so asymmetric that ill-proportioned pears occurred to their informed bodies, can demonstrate no second edible fruits.” A small Disney female duck figurine: “You were lecturing on changed eggs.” A small yellow Volkswagen beetle car toy: “Lemons, more eyeless than other beady sectors, would pardon no optical drops.”[6]

In further relegating the user to an ever-limited role with the physical, Rokeby increases the mental requirements of consciously interpreting the results of their interaction with the system and its interfaces. When confronted with the phrases output by The Giver of Names, one must ultimately deal with the formation of language and how our use of language defines ourselves. Yet unlike the highly visible system of The Giver of Names, we are largely and for the most part unaware of what defines language and what ultimate purpose it plays. In calling attention to these readily discussed issues, it can be inferred that Rokeby is wanting us to consider the construction and motives behind interfaces already codifying the way we interact with virtual spaces and the resulting definition of ourselves in the mirrored physical space.[7]

While these concepts are not new to his prior body of work, the uniting of both the physical and mental elements are innovative and show signs of where Rokeby is headed in future works. The user will be called upon to consider not only how the interface dictates the convergence of the physical and virtual, but also how the language and classification employed in society impacts the virtual spaces as well. The parallels forged between the discrepancies between the language of the physical and the coding in the virtual demonstrate his desires to move from mediation to a more transparent and discursive form of interfacing with technology and in effect ourselves. A moment where we may inform and enhance ourselves at the interstices of reality and the unknown both in the virtual and our very consciousness.

WORKS CITED:

Huhtamo, Erkki. “Silicon remembers Ideology, or David Rokeby’s meta-interactive art.” http://homepage.mac.com/davidrokeby/erkki.html. Pg. 5 of 6.

Manovich, Lev, The Language of New Media. MIT Press: 2001, pg 64.

Rokeby, David. “The Construction of Experience: Interface as Content.” Digital Illusion: Entertaining the Future with High Technology. ACM Press: 1998.

Rokeby, David. “Lecture for “Info Art,” Kwangju Biennale.” http://homepage.mac.com/davidrokeby/install.html. Pg 2.

Rokeby, David. http://homepage.mac.com/davidrokeby/gon.html.

Rokeby, David. http://homepage.mac.com/davidrokeby/src.html.

Rokeby, David. http://homepage.mac.com/davidrokeby/vns.html

---

[1] Manovich, Lev, The Language of New Media. MIT Press: 2001, pg 64.

[2] Rokeby, David. http://homepage.mac.com/davidrokeby/vns.html.

[3] Rokeby, David. “The Construction of Experience: Interface as Content.” Digital Illusion: Entertaining the Future with High Technology. ACM Press: 1998. Pg 6 of 15 of article.

[4] Rokeby, David. http://homepage.mac.com/davidrokeby/src.html

[5] Rokeby, David. “Lecture for “Info Art,” Kwangju Biennale.” http://homepage.mac.com/davidrokeby/install.html. Pg 2.

[6] Huhtamo, Erkki. “Silicon remembers Ideology, or David Rokeby’s meta-interactive art.” http://homepage.mac.com/davidrokeby/erkki.html. Pg. 5 of 6.

[7]Rokeby, David. http://homepage.mac.com/davidrokeby/gon.html