The Preservation of Interactive Multimedia Installations

Abstract. Interactive Multimedia Installations (IMIs) are invaluable for the history of art and culture, for media technology and for other disciplines involved in the scientific research and the areas of application. Despite their importance, Interactive Multimedia Installations show some weaknesses that threaten their chance to be preserved long enough to be studied or replicated: (1) they show a complexity that cannot be oversimplified; (2) they are highly refractory to preservation. Moreover, they show an alarmingly short life expectancy compared to other cultural materials, such as audio-video carriers, let alone paper documents, paintings and sculptures. This paper presents the main problems related to the preservation of Interactive Multimedia Installations, and proposes the core of a possible solution.

Introduction

The concept that multimedia refers to something more than juxtaposition of multiple media was introduced during the digital-machine driven revolution of the last six decades, where the term "medium" is not to be intended etymologically (otherwise, anything providing a multi-sensory stimulation would be multimedia, which is not the present technological-related meaning). In [14], the "new media" are configured according to two levels: a computer level (including data structures, file formats, functions and variables) and a cultural level (a classic novel, a favourite picture). These levels will prove useful in the conceptual model presented in Section 4. Just as the meaning of the term multimedia needs to be agreed upon, here the term 'interaction' holds a technological-related meaning -- and in this case the definition matches the etymology. As human beings, our experience of the world is basically inter-active and, in this sense, a condition we are most familiar with. The novelty of the concept resides in that, due to unprecedented computational power, machines are able to support interaction. More precisely, they support real-time multimodal interaction, with increasingly sophisticated design, inspired by the model of "ubiquitous computing" [7,16,19].


Figure. Hierarchical representation of the differences between old and new media, according to [14]. In Manovich's opinion, they are not absolute laws but "general tendencies of a culture undergoing computerization".

The article is organized as follows: Section 2 introduces the scope of preservation and Section 3 focuses on the problems associated with preservation. Section 4 describes the authors' proposed solution.

Preservation

Preservation is intended to grant unrestricted access to cultural materials, which must be made available "forever" -- decades or centuries, or long enough to be concerned about the obsolescence of technology [10]. An official document issued by UNESCO [8] reported that in 2002 seventy to eighty per cent of documentary heritage in Eastern and Central Europe was estimated to be inaccessible or in urgent need of preservation. The most authoritative institutions in the field agree that at present the most reliable technology for long-term preservation is digital, as it allows easy copying without loss of information. As a drawback, specific dangers such as failure without any warning should be considered, as well as the necessity to re-think concepts such as authenticity, ownership and copyright of the documents.

Synthesizing tangible and intangible objects into digital preservation copies (in [11], a preservation copy (or archive copy) is defined as "the artifact designated to be stored and maintained as the preservation master". It is an organized data set that contains all the information carried by the original document, accompanied by the metadata and the documentation about the preservation process) causes the multiplicity of documents (multimedia) to be encoded as "unimedia" [15]. Despite the fact that digital technology is considered future proof, the exponential pace of technological evolution brings about a shift from the traditional focus on degradation of physical media to the awareness of the risks of obsolescence of the stored data.

Interaction introduces an additional element of complexity to the problem of the preservation of Interactive Multimedia Installations, already raised by some musical "open works" such as Scambi (1957) by Henry Pousser. A systemic approach to preservation could overcome the limits of the current approaches, which require that objects are (made) static for preservation. In this sense, an analogy can be made between interactive multimedia systems and open systems. According to [1], every living organism is essentially an open system that maintains itself in a continuous inflow and outflow (a building up and breaking down of components) so long as it is alive. The commonalities with interactive multimedia systems are remarkable.
A good model for preservation should base on timedependent aspects of interactive systems, rather than distort the present ones to forcedly deal with it. Section 4 discusses a possible solution that considers these aspects.

Current trends of multimedia artistic works show an increasing attention for full-body interaction. According to the embodied approach to the study of music cognition [12], corporeal and nonverbal articulations may enable a paradigm shift in the preservation of cultural heritage, in particular of interactive processes. The crucial idea is that body gestures may hold meaningful significations with respect to the subjective experience of involvement with music, i.e., that body movement may provide efficient musical descriptors different from the traditional text-based descriptors, which are effective for some features but totally fail to attain others.

A particular class of works that should be considered for preservation consists of the replications, the reinterpretations and the virtual models that resulted from important recovery actions, some of which were financed by the European Union, such as DREAM (Digital Re-working/re-appropriation of ElectroAcoustic Music), 2010-2012, and VEP (Virtual Electronic Poem: VR-simulation of the Poème Électronique by Le Corbusier, E. Varèse and J. Xenakis (1958), 2002-2004).

Finally, a sucessful model for preservation would enable the creation of a common European repository, that institutions (e.g., archives) and artists could populate with descriptions, data, pictures, videos and testimonies. The model would also result in a set of guidelines for a consistent organization of the data within the repository. Artists would be encouraged to produce a documentation about their new works along the lines of the framework: thus, the very important goal of actively involving the artists in the process of preservation would be achieved, and the long-term storage planning would be optimised.


Figure 1. Characteristics of the multilevel approach to preservation. Each level (bits, data, record, experience) can be static/dynamic, physical/logical, and has a limited life expectancy. Levels are depicted on the x-axis, properties on the y-axis, and values on the z-axis. The properties static/dynamic and physical/logical assume only two values or states: lower for static and physical, upper for dynamic and logical. Colors refer to the bins associated with the life expectancy of each level expressed in years.

Problems related to preservation

Some of the problems regarding the preservation of Interactive Multimedia Installations fall within the scope of established research fields (e.g., audio preservation), where standards and methodologies are available; some are addressed by novel research fields, where best practices are in the process of being defined. But there is also a wide area of intersection where different disciplines are called to integrate opposite (sometimes conflicting) approaches, in order to achieve satisfactory solutions.

Most of the disciplines that made a contribution to the table of music research during the twentieth century (e.g., psychology, sociology, acoustics, physiology, neurosciences, cognition sciences and computer science) describe music through a different viewpoint, and they are interested in different aspects (e.g., the propagation of pressure waves in space, the harmonic relations, the capacity to evoke emotions). No matter how good the individual descriptions are, none will attain a comprehensive picture, neither will disciplines that are just put next to each other without much interaction. It is the same for Interactive Multimedia Installations, where music is only one of the constitutive elements. This calls for an approach that goes beyond the single disciplines, becoming "transdisciplinary" [13] -- making way for new professional figures, whose added value precisely resides in the ability of switching contexts.
Some of the main problems can be summarized as follows:

• Complexity (meaningful distribution over multiple media);
• Interaction (intrinsic and extrinsic);
• Physical deterioration;
• Obsolescence of:
• Hardware;
• Software environments;
• File Formats;
• Programming language;
• Absence of score, or notation;

Figure. Graphical expression of the multiple relations that should be made possible between the elements contained in the preservation copy of an Interactive Multimedia Installations. Arrows represent semantic configurations, graphically rendered as layers. Single configurations are showed in subfigures (b), (c) and (d). Paths and cycles of each layer never intersect.

Current cataloguing standards provide that documents are classified by homogeneous typologies and, accordingly, that multimedia works are dismembered and their components grouped by typology. Interactive Multimedia Installations come as a multidimensional "assembly of artifacts" [2], i.e., they consist in the combination of several partial documents, mainly sonic/musical and visual documents representing intangible contents with a cultural and/or social signification. The documentary unity must be (temporarily) violated for cataloguing and subsequently restored to deliver the information as a whole. This approach leads to a variety of information systems using different formats for data storage [4], and the low or absent interoperability among repositories makes the reconstruction of the documentary unity a problematic or impossible task, thus wasting or betraying the ultimate meaning of the work. Defined by a high typological variety, multimedia are particularly threatened by casual dismembering -- as any bundle of items whose informational potential resides in the relation of the parts.

A good model of Interactive Multimedia Installations intended for preservation should be able to support: (a) all approaches to preservation (i.e., future re-installation, historical philological studies, musicological analysis, informed fruition, ...); and (b) all approaches to restoration (i.e., documental, aesthetical, sociological and reconstructive).

The definition of a conceptual model for the Interactive Multimedia Installations is a necessary step for any operative action.

Proposed solution: a multilevel approach

One of the few works related to Interactive Multimedia Installations is the ontology approach used in [17] and [18] to describe them and their internal relations to support the preservation process. In [3], some of the authors proposed a functional categorization of documents as an extension of the CIDOC Conceptual Reference Model (CIDOC-CRM), an ISO standard for describing cultural heritage [5,6,9]. Starting from the assumption that the creative process is impossible to freeze, a possible solution may come from a combination of approaches, defined as distinct levels of preservation, not necessarily sequential, with different purposes and ways to be performed. Each level pursues different goals, although there might be overlapping contents. Two forms of preservation are considered: static, where records are created once and not altered, and dynamic, where records are changed and updated. Finally, levels may consist of a physical or logical content. Figure 1 shows each level and the associated properties.

Level 1. Preserve the bits: each part of the original installation that can be directly preserved: static, physical, life expectancy 5-10 years.
All the data are kept in the original format (the problem of their interpretation is a matter aside), and the risk of introducing alterations must be avoided.

Level 2. Preserve the data: technical notes, comments and useful information about the realization of the installation: dynamic, physical, life expectancy up to 30 years.
Includes a high level descriptions of algorithms and models.

Level 3. Preserve the record: any element that was modified or updated in respect of the original installation: static, logical, life expectancy 10-20 years.
It includes reinterpretation of the patches and information about the context.

Level 4. Preserve the experience: any document that bears witness to some aspect of the installation: dynamic, logical, life expectancy up to 30 years.
Includes hardware, software, interviews, audio/video recordings, usability tests of the original system, as well as information about people (composers, directors, performers, technicians) involved in the original performance and their roles.

It should be noted that some interactive sets provide that objects are used or consumed during interaction (food, perishable material, ...). In this case, one or more parts of the system may literally not exist after the process of interaction.


Figure 2(a). Interactive Multimedia Installation Piazza Pinocchio, exhibited at the Expo 2002 in Neuchâtel, Switzerland, from May to October 2002. Rear view of the massive wooden structure that housed the installations inspired by the tale of children's novel Pinocchio.


Figure 2(b). Interactive Multimedia Installation Piazza Pinocchio, exhibited at the Expo 2002 in Neuchâtel, Switzerland, from May to October 2002. One of the EyesWeb patches used during the exhibition, migrated from EyesWeb 2.4.1 to a new environment with EyesWeb XMI 5.1.0 on Windows XP.

Case studies

The approach described is currently being applied to a selection of Interactive Multimedia Installations by Carlo De Pirro (1956-2008: composer and professor at the Conservatory of Music "Venezze" in Rovigo, Italy, from 1982 to 2008) at the Centro di Sonologia Computazionale (CSC) of the Department of Information Engineering, University of Padova, Italy. More detailed description of the case studies are reported in [3], in particular: 1) the multimedia installation Piazza Pinocchio, exhibited at the Expo 2002 in Neuchâtel, Switzerland, from May to October 2002; and 2) the Carillon della Materia, currently exhibited at the Parco Collodi in Pisa, Italy.
Access the archive material about the Magic Room (Piazza Pinocchio) organized according to the model presented in this article.

Piazza Pinocchio consisted in a closed environment dedicated to children, like a magic room, in which each gesture turned into sound, image, color (cross-modal interaction). The visitors got involved in a communication of expressive and emotional content in non-verbal interaction by multisensory interfaces within a shared and interactive mixed reality environment. The system focused on full-body movements as primary conveyors of expressive and emotional content. The audio/video processing was performed by the EyesWeb open platform.
The work was dismantled after exhibition and never reproduced due to its massive architectural structure. The original equipment and the software tools were documented (level 1). When the preservation process started, some audio files employed during the original performance were missing. Algorithms and models were described (level 2), and the original EyesWeb patches were migrated to the EyesWeb XMI release and run on Vista OS (level 3). Interviews and photographic material are currently being collected and commented (level 4).

The Carillon della Materia is a wooden structure inspired by the tale of Pinocchio, where common tools of a carpenter's workshop turn into musical instruments, automatically controlled according to a musical score especially written by De Pirro. The wooden frame of the Carillon was damaged during the transfers to other exhibitions, and it was subsequently restored and reinforced (level 1). Interviews and photographic material were collected (level 4). Electronic controls were made on purpose and not maintained through the years, so today the Carillon is a silent piece of sculpture: the people with the knowledge about its original configuration are not available, and no video recordings were made to document the way the Carillon was supposed to move and sound in the author's intention. This example shows that the preservation of the physical components is nullified, if the knowledge to make a sense out of them is lost.

Conclusions

The main problems related to the preservation of Interactive Multimedia Installations were introduced. Effective methodologies for preservation are an urgent matter because the works' life expectancy is alarmingly short compared to other cultural materials. To date, acceptable methods to represent and preserve complex digital entities that contain combinations of text, data, images, audio, and video, and that require specific software applications, do not meet the requirements for an effective preservation of Interactive Multimedia Installations. Moreover, current cataloguing standards provide that documents are classified by homogeneous typologies and, consequently, that Interactive Multimedia Installations are dismembered for storage, which may result in the impossibility to reassemble the work as a whole unless protocols of interoperability among repositories are defined.
The authors presented the core of a possible solution, consisting in a multilayered approach, able to support different goals for preservation (documenting, academic studies, ...) and re-installation (new exhibitions, interpretations, tributes, ...).

References

1. von Bertalanffy, L.: General System Theory. George Braziller, New York, NY (1968)

2. Bowers, J., Bannon, L., Fraser, M., Hindmarsh, J., Benford, S., Heath, C., Taxen, G., Ciolfi, L.: The Disappearing Computer, LNCS, vol. 4500, chap. From the Dis- appearing Computer to Living Exhibitions: Shaping Interactivity in Museum Set- tings, pp. 30-49. Springer-Verlag Berlin Heidelberg (2007)

3. Bressan, F., Canazza, S., Roda, A., Orio, N.: Preserving today for tomorrow: A case study of an archive of interactive music installations. In: Workshop on Exploring Musical Information Spaces (WEMIS) in conjunction with ECDL 2009, pp. 24-29. Corfu (Greece) (2009)

4. Canazza, S., Dattolo, A.: The past through the future: A hypermedia model for handling the information stored in the audio memories. Journal of New Music Research 38(4), 381-396 (2009)

5. Doerr, M.: The cidoc crm - an ontological approach to semantic interoperability of metadata. AI Magazine 24(3), 75-92 (2003)

6. Doerr, M.: Increasing the power of semantic interoperability for the european li- brary. ERCIM News 26, 26-27 (2006)

7. Dourish, P.: Where The Action Is: The Foundations of Embodied Interaction. MIT Press (2001)

8. Edmonson, R.: Memory of the World: General Guidelines to Safeguard Documen- tary Heritage. UNESCO (2002)

9. Gill, T.: Building semantic bridges between museums, libraries and archives: the cidoc conceptual reference model. First Monday 9(5) (2004). URL http: //firstmonday.org/issues/issue9_5/gill/index.html

10. Hedstrom, M.: It's about time: Research challenges in digital archiving and long- term preservation. Final report on a workshop on research challenges in digital archiving. Tech. rep., Library of Congress, Washington, DC (2002)

11. IASA: The IASA Cataloguing Rules. IASA Editorial Group (1999)

12. Leman, M.: Embodied Music Cognition and Mediation Technology. MIT Press (2008)

13. Leman, M.: Systematic and Comparative Musicology: Concepts, Methods, Find- ings, Hamburger Jahrbuch fur Musikwissenschaft, vol. 24, chap. Systematic Musicology at the Crossroads of Modern Music Research, pp. 89-115. Peter Lang Verlagsgruppe, Frankfurt am Main, Germany (2008)

14. Manovich, L.: The Language of New Media. MIT Press (2001)

15. Negroponte, N.: Being Digital. Vintage Books, New York, NY (1995)

16. Ng, K., Pham, T.V., Ong, B., Mikroyannidis, A., Giaretta, D.: Components of interactive multimedia performance and preservations. In: Electronic Visualization of the Arts (EVA) London Conference (2007)

17. Ng, K., Pham, T.V., Ong, B., Mikroyannidis, A., Giaretta, D.: Preservation of in- teractive multimedia performances. International Journal of Metadata, Semantics and Ontologies 3(3), 183-196 (2008)

18. Ng, K., Pham, T.V., Ong, B., Mikroyannidis, A., Giaretta, D.: Metadata and Se- mantics, chap. Ontology for Preservation of Interactive Multimedia Performances. Springer US (2009)

19. Rocchesso, D., Bresin, R.: Emerging sounds for disappearing computers. In: The Disappearing Computer, pp. 233-254 (2007)