..a space filled with data

The concept of Mixed Reality (MR) unfastens real-time interactivity from immersive environments and integrates media participation technologies into processes which are localised in real space. MR thus becomes a method by which real physical spaces and electronic, intangible data spaces may be overlaid in space and time. The conceptual starting point for this is the multi-layered interweaving of different reality levels which function as reference systems in managing reality. The psychologist Hinderk Emrich states »that human beings are in fact always living in a state of Mixed Reality which we ourselves continually condition and construct«.

The use of Mixed Reality technologies is thus developing processes of perception and cognition which are located in both real and virtual space at the same time.

A central element of the MR strategies presented here is to provide user-friendly interfaces and to allow communication with distant partners using everyday means of expression such as speech, gestures, touch and movement. The use of interfaces which fluently integrate established communication practices already in every day life (both direct and technical) into complex media communication systems allows intuitive navigation and interaction in the data space.

Tracking, position detection and image recognition systems can be used as public interfaces with displays in public data spaces. Personal interfaces such as cellular phones or palmtops can be further developed as interfaces with the public data space.

sketch of mixed Reality test arrangement {enlarge}

eMUSE
The Electronic Multi-User Stage Environment (eMUSE) is a software platform for fusing virtual and real action spaces. The audio-visual and network modules of the eMUSE architecture support scenarios networked in real time and based on Java and the VRML browser. The MARStrack Vision System transmits the position data (x/y coordinates and the gestures of the users) to the eMUSE server which in turn transmits this data to several clients. As well as including several persons in a communal real space and depicting this in a virtual environment, eMUSE also allows the participation of spatially distant users via the Internet.

The platform provides important components of the performance arts and links them with interactive and networked scenarios - the possibility of free movement in space, the transference of that movement in real time into a virtual environment, the participation of several performers and their interaction with each other. One of the first uses was in the »Murmuring Fields« installation at Transmediale 1999 in Berlin. This investigated the practicability of the interface in a public space with a high fluctuation and a heterogeneous user group. The presentation was accompanied by an evaluation of user behaviour to assess the possibility of its use in functional as well as artistic contexts.

communication flow and telepresence linking real and virtual space {enlarge}

Murmuring Fields
At the presentation at Transmediale 1999, a projection area divides the space into an action space and an audience space. The two-dimensional image carrier stretches practically the whole height and width of the space and shows a virtual scenario. The structure of this computer-generated backdrop stimulates physical activity. It consists of sketchy, drawn figures which, through their organic design, recall the dynamics of physical movement. If the visitor steps into the projection area, he intuitively begins to interact with the virtual world. A camera on the ceiling recognises the position of the user and represents his movements using an avatar in the virtual space which adjusts its perspective to correspond to the actual position of the user in real time.

The simultaneous presentation of bird's eye views and subject's eye views gives the user alternating reference layers which explore the dynamic relationship between media immersion in and physical distance from the virtual space.

{[1] enlarge} {[2] enlarge}

The MARStrack Vision System supports an interface which allows maximum free movement. Thus, the media space is not defined in terms of an immersive environment as a surrogate for real space, but is overlaid with real space. One element which combines these two areas is the interaction of the user and his intuitive movements in real space.

The interactive event is accompanied by a soundscape which generates sound collages dependent on the movements and gestures of the user in real space. This adds an acoustic dimension to the visual scenario in the projection area.

At the »Awareness of reality, reality of awareness« symposium in Bochum in 1999, organised under the umbrella of the »Figure Theatre of Nations« (FIDENA) festival, the aesthetic potential of Murmuring Fields was investigated as an experimental mixed reality theatrical form.

In this version of Murmuring Fields, the interactive sound space is the key element of the scenario. The acoustic space consists of philosophical quotations on the influence of media on the living world which are transferred into the sound space where they are fragmented down to a phonetic level. The sound generation focuses the audience's attention on the real actors. Thus the installation localises the centre of the action not in a virtual, visually transmitted image space but in the dynamic field of physical bodies, the real stage space and the intangible atmosphere of the interactive soundscape.

map of Murmuring Fields soundscape {enlarge}

Both versions of Murmuring Fields investigate the possibilities of using the eMUSE system in an artistic context. The aim of further development is to utilise the central element of the intuitive interface and the possibility of networking for further functional contexts, in order to develop interactive practices to realise the paradigm of the mixed reality space in everyday scenarios.

i2tv - a platform for interactive broadcast formats and live productions
i2tv (interactive Internet TV) is platform for media productions integrating Internet participants with participants on site. The challenge is to create an environment in which Internet participants and local audience become equal partners, inspite of their different situations (on-line, on-site).To allow this the system combines 3D Internet environments with digital television and mobile communication interfaces.

i2tv stimulates and explores forms of non-linear dramaturgy through communication concepts which involve the individual user in a media- connected community through situated action.

i2tv layers of participation {enlarge}

The audience,both local and on the Internet,can not only follow the event,but also actively shape it,using text,image,live audio or video. The different viewpoints of the participants become instruments that shape the dramaturgical structure.

In a prototype Mixed Reality television game based on i2tv, Ernst Jandl's phonetic poem »Ottos Mops« is split into single words as a starting point for a collaborative word-sound collage. Participants on-site, in Internet cafés or outside in the street,are equipped with a mobile phone,palmtop,PC or touch screen,in order to send their word fragments of the poem into the Virtual Studio at GMD. The moderator in the studio plays a conductor who binds the incoming words of the networked audience into a new phonetic poem.

The i2tv experiments investigate the specific factors of situations that integrate networked participants in a real-space event - how can the conflict between immediate involvement in an interactive situation and distance for reflection be turned into fruitful ground for new concepts? how must spatial situations for on-line and on-site participants be organised, in order to communicate to both groups a feeling of presence?

coming soon: Mixed Reality TV

The presented examples depict experimental arrangements in which different levels of the virtual and the real intervene into each other. The examination of fundamental questions of spatial experience and its integration into experimental contexts stimulates the development of media communication structures which regard the data space not as a surrogate of real space but as an integrative component of the living world. A key part of the research involves the conception and development of transfers between these different action levels - reality and virtuality are not regarded as exclusive living worlds, but as interacting levels of human activity.