Pockets Full of Memories

Inhaltliche Beschreibung

1.Project Description and Database Archive

"Pockets Full of Memories" (PFOM) integrates the real space environment of a museum installation with virtual access to the database through the internet. Visitors to the exhibition contribute visual and descriptive information to the digital archive about an object in their possession at the time of their visit. The data contribution takes place in the entrance area where the public inte racts with a kiosk-like scanning station in a two-step process that consists of the scanning of the object followed by filling out an questionnaire through a touchscreen interface to describe its attributes. The image of the object and descriptive data are then stored in a database that grows throughout the duration of the exhibition. Presentation and access to the database occurs both online and in a large-scale projection in the gallery space of the museum.

The archive of objects consists of objects that museum visitors carried with them, for instance, such common items as phones, keys, toys, fragments of clothing, personal documents, currency, reading material, and others. The size of the scanning box was the only limiting factor that determined what could be added to the archive. The expectation in the early stages of planning was that the majority of contributions would consist of everyday common objects and the final result would provide an overview of the range of things people carry with them. Nonetheless there was the hope that some members of the public would be creative in their contribution choices, methods and descriptions and in fact there are a few objects that go beyond the everyday, for instance a marriage proposal note. We were also aware that such public interactive systems are testing situations to a segment of the audience who want to see how robust such systems are by trying to break through both the technical and conceptual limits of the project. One of the unexpected contributions to the archive included the numerous scans of body extensions such as objects: heads, hands and feet which increased exponentially once the initial examples entered the archive. The overall quality of the archive is a consequence of the dialogue that occurs between the audience's perception of the archive's holdings followed by a contribution that functions as a statement of participation and an engagement to leave a trace behind once the public has exited the museum. Contributing an image and descriptive information of one's personal object into an institutional environment such as a museum or database or archive certainly provides a sense of fulfillment but the best trace left behind seems to be a direct image of one's body parts. The many scanned heads, hands and feet have augmented the archive from simply being a collection of objects to encoding it with the corporeal presence of the contributors transforming the digital data environment into a metaphoric extension of the human body.

2. Ordering Data and the SOM Algorithm

The database archive of objects is ordered by the Kohonen Self-Organizing Map prior to being projected in the gallery space and accessed online. Due to current bandwidth limitations and scale of the gallery projection, the map of objects has been limited to 280 objects, so a selection is first made out of the total database each time the SOM is activated (once per minute). The selection is based on a percentage sampling of the database's life with priority given to the ten most recent entries. Honkela describes the function of the SOM in this project as follows:
"The Self-Organizing Map (SOM, also called Kohonen map) algorithm is the basic method that is used to create the "wall of objects". The SOM organizes the input items (contributed objects by the public) into an ordered display, a 2 dimensional map. On the map two items tend to appear close to each other if they have similar input features. In this exhibition, the input features consist of attributes and keywords. The attribute values and keywords are given by the exhibition visitor. They are transformed into numerical form that can serve as inputs. The algorithm may start from a random state of the map. Through the process of iteratively (repeated recalculation) handling the inputs it reaches an ordered state.
The map consists of a collection of map nodes that can be thought as places on the map landscape. On the map, nearby nodes tend to have similar items. Close to each there may be items that have been given similar attribute values, or items that have been named similarly. Thus, all the items with a particular keyword are not necessarily next to each other if the other features vary. Moreover, even if the visual qualities of an image are very similar, it may very well happen that two persons evaluate the item very differently based on their subjective point of view. There are also cases in which even the neighbors are rather far from each as there are occasionally dividing valleys and mountains ranges also in the natural landscapes.
The order of the final map is a consequence of all the inputs. The phenomenon is called emergence: the order is not determined beforehand. The order emerges through the audience contributions. The classification system is not specified by hand but it is created through the large number of local interactions on the map. This is why the system can be called 'self-organizing'. Metaphorically, similar items look for each other without any centralized command.
The principle of the SOM was developed originally by academician Teuvo Kohonen. The inspiration for this innovation has stemmed from the numerous neurophysiological studies in which it was shown that in the cortex of the brain similar kind of maps can be found. Perhaps this is a reason why the method seems to be appealing in several application areas." (Timo Honkela letter to the curator Boris Tissot)

3. Semantic Meaning and Linguistic Description

Because the positioning of an object in the SOM map is completely dependent on the way the contributor describes them, the emphasis in this work shifts to the function of linguistic description and semantic interpretation as opposed to the object's immediately recognizable literal or physical properties. Contributors engage in a creative process when they prescribe new keywords and evaluate their object according to the questionnaire's attributes. Even though the map may contain numerous cell phones, watches and hands, their positioning across the map reflects the contributor's perception of them and the choice of words used to describe them. One of the key experiences of the exhibition is to watch the positioning of the object's image on the large-scale projection. After a short wait following data entry, the image appears on the screen highlighted with an orange frame to make it easier to identify. As the SOM processes the data every minute, it scans each line of the map and replaces or moves objects based on the new order. This process goes on continuously and provides the opportunity for comparison and reflection about the descriptive choices the contributor has made to define the object. Accessibility on the internet further enhances study of the map as each object can be clicked on to see its properties and attached stories, resulting in comparisons to surrounding neighbors. Internet interaction has provided another means by which to extend the dialogue for visitors, as the museum and internet audience have the opportunity to add comments and stories to any object, and from anywhere in the world. Many visitors who have traveled from other geographical areas have used this as a means to make contact with friends and family back home who then have added their own responses.

4. Visual Identity, Graphic and Interface Design

All graphic design aspects of the PFOM project have been designed by the Projektriangle team as a result of extensive dialogues about the meaning and function of all of the elements: the signage, the interactive questionnaire, the installation and website. The information environments within which data is organized and accessed impose a meta-level of meaning that redefine the content passing through it, much of which is implicit and necessitates extensive study of its functioning. I was very much interested in handing over the design solutions to Projektriangle as their work places an emphasis on coherent systems and the development of a specific visual vocabulary determined by the needs of the project. The thematic repetition throughout all the design elements focused on the process of the production of meaning beginning with the public's personal contributions and descriptions which become synthesized into an institutional archive. The signage in the exhibition space floor exemplifies the three-phase transformative process from object to descriptive parameters to data. From the entrance to the scanning station the floor is covered by graphic icons of objects physically guiding the public's movements to the various stages of the installation. Between the triangular space from the scanning station to the web viewing stations, the floor markings have changed to keywords which are also listed on the large wall to the right of the projected map. The markings in front of the final stage of viewing: the projected map, consist of id numbers. (George Legrady)

• › Realizing a Digital Media Installation [PDF | 139 KB ] [link 02]
• › system-setup [JPEG | 348 KB ] [link 03]
• › system-setup (detail) [JPEG | 88 KB ] [link 04]
• › input_station [JPEG | 126 KB ] [link 05]
• › db_som [JPEG | 98 KB ] [link 06]
• › visualisation [JPEG | 67 KB ] [link 07]
• › draft [PDF | 72 KB ] [link 08]

Technische Beschreibung

The public contributes a digital image and description of an object they carry with them at the time of the exhibition.
- The description is entered through a questionnaire
- This data is stored in a database that grows during the life of the exhibition
- The data is then organized by an algorithm that groups the objects according to proximity based on their descriptions
- The algorithm is titled a "self-organizing map" and its function is to map out in a 2 dimensional space the relative closeness of each object to the group based on their properties
- The spatially organized map is then projected on a large wall in the gallery space and can also be viewed on the internet.
- The public can explore and view the data for each object and related objects through terminals placed in the exhibition space.
- Audiences both in the exhibition and on the internet, can additionally contribute by adding comments and stories to the objects.
- The archive of objects, in the end, also becomes a site for the collection and exchange of stories.

The self-organizing map was produced in Helsinki by Dr. Timo Honkela and his team from the Media Lab (University of Helsinki) working in neural-net based algorithms.

"The Self-Organising Map (SOM, also called Kohonen map) algorithm is the basic method that is used to create the "wall of memories". The SOM organises the input items into an ordered display, a 2 dimensional map. On the map two items tend to appear close to each other if they have similar input features.
In the exhibition, the input features consist of some attributes and keywords. The attribute values and keywords are given by the exhibition visitor.
They are transformed into numerical form that can serve as inputs. The algorithm may start from a random state of the map. Through the process of iteratively (repeated recalculation) handling the inputs it reaches an ordered state.
The map consists of a collection of map nodes that can be thought as places on the map landscape. On the map, nearby nodes tend to have similar items.
Close to each there may be items that have been given similar attribute values, or items that have been named similarly. Thus, all the items with a particular keyword are not necessarily next to each other if the other features vary. Moreover, even if the visual qualities of an image are very similar, it may very well happen that two persons evaluate the item very differently based on their subjective point of view. There are also cases in which even the neighbors are rather far from each as there are occasionally dividing valleys and mountains ranges also in the natural landscapes.
The order of the final map is a consequence of all the inputs. The phenomenon is called emergence: the order is not determined beforehand. The order emerges through the audience contributions. The classification system is not specified by hand but it is created through the large number of local interactions on the map. This is why the system can be called 'self-organizing'. Metaphorically, similar items look for each other without any centralised command.
The principle of the SOM was developed originally by academician Teuvo Kohonen. The inspiration for this innovation has stemmed from the numerous neurophysiological studies in which it was shown that in the cortex of the brain similar kind of maps can be found." (Timo Honkela)

Technical comment on the Self-Organizing Maps and their first implementation (Media Lab group):

- For generation of maps we used sompak reference inplementation, which can be obtained freely from Helsinki University of Technology, where SOM's were invented.
- Feature vectors consisted of simple scalar values obtained from straight from visitors input with rather long section wich was delivered from the textual input by random vector mapping introduced by Mr. Honkela in his doctoral thesis, which is available from net.
- For the labelling of the map we developed a new tool to avoid overlapping of images and to give emphasis on fresh entries.
- There were no sophisticated databases implementations on our part, everything (maps, input data, dictionary for random vectors ) was stored in textfiles and managed by bunch of perl-scripts.

In the end, and in contrast to the original plans, the work (visual image, interactivity, etc.) was quite a compromise:
- Projection was rather small in physical size and contained small number of items.
- Planned PDAs with wireless connection for visitors which would allow individual exloring of the map and annotation of objects with stories or "memories" was dropped.
- I didn't check out how well different objects finally organized.
(Timo Koskenniemi)

Hardware / Software

2 to 4 screen projections, scanning station, 2 terminals, wall design

In 2003, PFOM I was updated to PFOM II: a digital scanning of objects replaced the former analog video scan and moreover the visualization was changed to 4 views: the image of the objects, a textual description of the each object, the movement of objects on the screen as they get organized, the u-matrix cluster mapping of the SOM algorithm which shows how the SOM organizes the objects in groupings.

Statement

Planning of "Pockets Full of Memories" (PFOM) began in the spring of 1999 as a result of a meeting with Boris Tissot, exhibition coordinator from the Centre Pompidou, who was interested in an exhibition that would integrate issues related to the intersections of memory, the archive, digital technology and the general public. A collaboration was developed with Dr. Timo Honkela from the Media Lab, University of Art and Design in Helsinki through CIRCUS (Content Integrated Research in Creative User Systems), a research and development project funded by the European Union under the Esprit program. Timo Koskenniemi from UIAH Media Lab contributed the implementation of the Self-Organizing Map algorithm. Kohonen's algorithm applied in a manner outlined earlier in Honkela's research (Honkela 1997) became a key conceptual framework for the PFOM project.
Production began in the summer of 2000, following funding by a Daniel Langlois Foundation for Art, Science & Technology grant and continued until the opening of the exhibition in April 2001. In addition to the contribution by the UIAH Media Lab team (Prof. Timo Honkela, Timo Koskenniemi and Petri Saarikko), an international team of specialists realized various components of the project. Marton Fernezelyi and Zoltan Szegedy-Maszák, from the C3 Center for Culture and Communication in Budapest designed and constructed the scanning station unit consisting of a touchscreen interactive software, an image digitizing system with image processing software, the database and networks software and the interactive station structure itself. The exhibition's visual identity, which included the exhibition design, scanning station designs, interface design for the questionnaire and website, was produced in Stuttgart by the Projekttriangle design team consisting of Danijela Djokic, Martin Grothmaak, and Jürgen Späth. The internet site providing access to the database and SOM algorithm was developed by Andreas Engberg at the CREATE Research Lab University of California, Santa Barbara. Dr. Brigitte Steinheider from the Fraunhofer Institute for Industrial Engineering in Stuttgart contributed with the development of the data description questionnaire and analyzed the data of the archive from a social science perspective. In addition to this work that examines the cultural and social nature of the collected information, she is also studying the complexity and impact of multinational and multidisciplinary teamwork practice in the production of the PFOM project.

The first prototype of "Pockets Full of Memories" was shown and beta tested at the ISEA (Inter-Society for electronic Art) in Paris in 2000. The installation with the accompanying Web site premiered at the Centre Georges Pompidou in Paris in 2001. It was featured 2003 at the DEAF Festival in Rotterdam, Ars Electronica in Linz, the "Aura" exhibition in Budapest and 2004 at the Kiasma Museum in Helsinki.

Veröffentlichungen

• Legrady, George: "Intersecting the Virtual and the Real: Space in Interactive Media Installations." In: T. Murray, (Hrsg.): Digitality and the Memory of Cinema, Wide Angle, Vol 21, No.1, Ohio University School of Film, 2000, S.104-113.

Sekundärliteratur

• Steinheider, Brigitte und George Legrady: "Realizing a Digital Media Installation: Problems and Synergetic effects of an Interdisciplinary Collaboration." In: Proceedings of MTAC 2001, Multimedia technology and applications, UC Irvine 2001, S. 255-260.
• Kohonen, Teuvo: "Self-Organizing Maps." Springer Series in Information Sciences, Vol. 30, 1995.
• Honkela, Timo: "Learning to Understand - General Aspects of Using Self-Organizing Maps in Natural Language Processing." In: D. Dubois (Hrsg.): "Proceedings of the CASYS'97, Computing Anticipatory Systems", American Institute of Physics, Woodbury, New York, 1997, S. 563-576.