Research Areas
- Audio Documents
- Cultural Heritage
- Multimodal Interaction for Well-Being
- Computational Creativity
- Affective Computing
- Acoustics for Security
- New Music Research
Audio Documents: Preservation and Valorization
Millions of hours of recorded music, sounds, voices, evidence of past life are being lost. The erosion of collective memory is due to the intrinsic physical and chemical instability of audio media, which results in a very short Life Expectancy (LE) (from a few years to a few decades), in sharp contrast with the LE of other cultural materials, such as paintings or sculptures, the degradation of which is measured in centuries or millennia.
At CSC, an innovative - philologically informed - methodology for the conservation and restoration of sound memories (speech archives and electroacoustic audio documents in particular) was defined, already applied in international research projects funded by large archives (e.g., Archivio Luigi Nono, Centro Studi Luciano Berio, Paul Sacher Stiftung, Arena di Verona, Scuola Normale di Pisa, Teatro Regio di Parma).
Our methodology considers the cultural context in which the document was produced with the help of an adequately equipped infrastructure (professional and functioning replay device, compatible with the format of the documents to re-mediate; analog-digital converter; precision incubator for the thermal treatment of magnetic tapes; etc.). In addition to a thorough and rigorous documentation, our methodology stands out for the high degree of automation achieved thanks to specifically developed software tools (discontinuity detection and equalization recognition) based on Artificial Intelligence, and for the systematic introduction of chemical-mechanical analyses aimed at identifying the most adequate treatment for the document.
Traditional (digital) access tools are inadequate: the fruition is untrue to the peculiarities of the (analog) original of that time, and it is also incomplete when the copy does not allow access to ancillary information. At CSC, several tablet, smartphone and web-based apps are designed and developed, aiming at recreating the experience of the original analog equipment. Such apps show a rich set of metadata about each piece of music, both textual (e.g., author, year, country of origin) and multimedia (e.g., pictures of the original media, video of the original tape running in sync with the audio). The virtual devices are controlled via a skeuomorphic UI reproducing in detail the behavior of a real device (see video below).
Sergio Canazza @WellCAM (University Museums Centre, CAM – Centro di Ateneo per i Musei), May, 8th, 2020. Quelle voci poco fa: l'ingegneria informatica per contrastare l'eclisse delle memorie sonore (in Italian):
Sergio Canazza @Istituto di Scienze del Patrimonio Culturale (ISPC, CNR), March, 3rd, 2021. Non di sola carta: l'ingegneria informatica a contrastare l'eclisse delle memorie sonore (in Italian):
References:
- Canazza, S., Schubert, E., Chmiel, A., Pretto, N., Rodà, A. (2022) The Magnetic Urtext: Restoration as Music Interpretation. Front Psychol. 13(22), https://doi.org/10.3389/fpsyg.2022.844009.
- Pretto, N., Fantozzi, C., Micheloni, E., Burini, V., & Canazza, S. (2019) Computing Methodologies Supporting the Preservation of Electroacoustic Music from Analog Magnetic Tape. Computer Music Journal 42(4), 59-74. https://dx.doi.org/10.1162/comj_a_00487
- Canazza, S., Fantozzi, C., & Pretto, N. (2015) Accessing tape music documents on mobile devices. ACM Transactions on Multimedia Computing Communications and Applications 12(1), 20 pages.
Computing and Cultural Heritage
Music is intangible and therefore its heritage is part of intangible cultural heritage. UNESCO (in: Convention for the safeguarding of the intangible cultural heritage. UNESCO, Paris, 2003) defines the intangible cultural heritage as “the practices, representations, expressions, knowledge, skills – as well as the instruments, objects, artifacts and cultural spaces associated therewith – that communities, groups and, in some cases, individuals recognize as part of their cultural heritage”. The objects and carriers related to music performance can be considered as part of this intangible cultural heritage, e.g. scores, musical instruments, costumes, posters, audio recordings, etc.
There are several kinds of musical instruments, each with peculiarities that have to be preserved and communicated. However, all of them share a common characteristic: they have to be played to be understood.
The multidisciplinary work carried out at CSC and concerning the valorization of the antique (e.g., a Pan flute) and modern (e.g., Studio Fonologia Musicale della RAI di Milano) instruments brings important outcomes. The first one is a methodology to develop multimedia installations to communicate and valorize musical instruments considering their cultural context. The multimedia installations can be a valid mean to provide an interaction with an artifact, that is usually not touchable or playable in museums. The interaction model utilized to provide access to the general public leverages on a multisensory (visual, auditory, tactile) interplay that includes both contextual information and a virtual counterpart of the artifact. The methodology includes an adaptation of Design Thinking to the context of interactive museum installations, and a matching design process that is deeply interdisciplinary.
Born in the first decades of the XX century, installation art was able to foresee the potential of new media, and it largely explored their possible applications. Today the impact of installation art on the contemporary artistic production is acknowledged worldwide. However, the deep interconnection with technology is taking its toll in terms of fast obsolescence, which may soon become irreversible loss.
The new problems raised by the interactive multimedia installation with respect to preservation are studied at CSC: not only these problems change over time, but they change according to what happens in their environment. Time is also a crucial factor as the multimedia installations are characterized by a life expectancy (LE) that is significantly lower than that of other cultural materials.
References:
- Bressan F., & Canazza, S. (2014) The challenge of preserving interactive sound art: a multi-level approach. International Journal of Arts and Technology, 7(4), 294–315.
Multimodal Interaction for Learning and Well-Being
In the last years, applications based on large-scale responsive environments have risen up as a convincing aid for learning simple as well as complex concepts in a playful way. The full-body interaction that characterizes these environments supports different learning styles and it is particularly fit for inclusion of people with disabilities, due to its richness in emotional engagement and ease of use. The CSC staff developed several serious games on a large-scale responsive environment, devoted both to music teaching and training of blind children. Experimental results showed a great user engagement and a satisfying amount of successful achievements in formal task activities.
Moreover, considering physical actions as an integral part of cognition, using the technological augmentation, we intend to promote learning activities that stimulate children to action. Technological augmentation in children serious games uses several sound and video events to make the children interact with the physical world around them.
References:
- Mandanici, M., Altieri, F., Rodà, A. & Canazza, S. (2018), Inclusive sound and music serious games in a large‐scale responsive environment. British Journal of Educational Technology, 49(4), 620-635.
- Mandanici, M., Rodà, A., & Canazza S. (2017). Bodily interactions in motion-based music applications. Human technology, 13(1), 82-108.
- Zanolla, S., Canazza, S., Rodà, A., Camurri, A., & Volpe, G. (2013). Entertaining Listening by means of a Technologically Augmented Physical Environment. Entertainment Computing, 4, 213–220, ISSN 1875-9521.
Computational Creativity
Computational creativity tries to obtain creative behaviors from computers algorithms; in particular, the CSC staff is interested in the creative aspects related to music composition and performance. The first computer algorithms aiming at generating music date back to the '60s, although there are examples of algorithmic music that even predate computers. Since then, a plethora of software for the generation and performance of music has been designed, using several rule-based and data-driven methods. Recently, the deep learning approach has opened new perspectives and has produced interesting results although two key questions remain almost unanswered: what does it mean for a machine to be creative? And, more pragmatically, how can one assess whether an algorithm is more or less creative?
References:
- Carnovalini, F. & Rodà, A. (2020) Computational Creativity and Music Generation Systems: An Introduction to the State of the Art. Frontiers in Artificial Intelligence 3(14). 20 pages.
https://www.frontiersin.org/articles/10.3389/frai.2020.00014/full - Carnovalini, F. & Rodà, A. (2019) A Multilayered Approach to Automatic Music Generation and Expressive Performance. 2019 International Workshop on Multilayer Music Representation and Processing, Milano, Italy. 41-48. IEEE.
- Simonetta, F., Carnovalini, F., Orio, N., & Rodà, A. (2018). Symbolic Music Similarity through a Graph-Based Representation. Proceedings of the Audio Mostly 2018 on Sound in Immersion and Emotion, Wrexham, UK. 26 pages. ACM.
Affective Computing
In affective computing, research on expressive music performance was initially oriented to analyze many musical performances played with different expressive intentions and to look for the relationships between measurable parameters and the various intentions, in order to understand what the strategies employed by the performers were. These analyses allowed to develop computational models to rendering and processing expressive content in multimodal interactive systems. The CARO model provide the user with the ability to modify the expressive content of a performance in a gradual way, both at the symbolic and at the signal level, applying a smooth morphing among performances with different expressive content, and adapting the audio expressive character to the user desires.
As a further example of how an expressive interpretation can change a melody, here is a Beethoven sonatina without any expressive intention:
And here is the same sonatina performed by the CaRo system:
References:
- Canazza, S., De Poli, G., & Rodà, A. (2015) CaRo 2.0: An Interactive System for Expressive Music Rendering. Advances in Human-Computer Interaction, 2015, Article ID 850474, 13 pages. https://doi.org/10.1155/2015/850474.
- Rodà, A., Canazza, S., & De Poli, G. (2014) Clustering affective qualities of classical music: Beyond the valence-arousal plane. IEEE Transactions of Affective Computing, 4(October-December), 364–376.
Acoustic Analysis for Security
Video-based surveillance systems may benefit from integration with other types of sensors. In particular, audio sensors can provide a major improvement because – unlike normal cameras – they are omnidirectional and do not require a direct line-of-sight with the sound source. Such capabilities can nicely complement vision to help localize an interesting or a dangerous event in the monitored area. Audio-based systems are able to locate a sound source in a room and at a short distance (few meters) using microphone arrays and signal processing techniques, generally based on time delay of arrival estimation (TDOA) and delay-and-sum beamforming (SRP). Compared to vision, the use of audition in surveillance system is in its early stages. The application of the localization techniques to the monitoring of large areas, such as squares or parks, poses difficulties not yet fully addressed. In general, the error with which a microphone array can localize a sound event varies, with the distance and the angle of the sound source with respect to the array position and depends on a number of design factors such as the shape of the array, its size, the distance between the microphones, the sampling frequency, the acoustic response of the environment, the presence of competing sound sources.
References:
- Salvati, D., & Canazza, S. (2013). Adaptive time delay estimation using filter length constraints for source localization in reverberant acoustic environments. IEEE Signal Processing Letters, 20(5), 507-510.
- Salvati, D., Rodà, A., Canazza, S., & Foresti, G. L. (2010). A real-time system for multiple acoustic sources localization based on ISP comparison. In Proc. Conf. on Digital Audio Effects, 201-208.
New Music Research
In the '60s CSC started as a lab for electro-acoustic music production. Today scientists, researchers, and technicians still collaborate with artists using the new art-science-interaction lab and our know-how as a support for the innovation of expressive forms in music and interactive multimedia arts.
Listen to Consonantico (by Alvise Vidolin and Giovanni De Poli, 1975), the very first computer music composition of the CSC:
A video version (by Sergio Canazza and Alvise Vidolin, 2021): https://youtu.be/szbkdkthmY8
The CSC aims at promoting and encouraging the production of works which use computer systems to control and to create music, especially when the projects make use of the technologies developed in our labs.
The artists working on musical projects at our labs are inspired by the opportunities that CSC technology allow them to do, and they may get scientific expertise on such matters by our researchers. Moreover CSC may help them organise concerts.
Today, if a musician wants to produce technology-enhanced music, s/he can do it with affordable commercial tools. But, if a composer aims at creating music that has never been heard before, then s/he will need something more than that. Turning to CSC, s/he may make a well-informed use of our labs and get the most out of our researchers' knowledge and competence. At CSC we have the right expertise to develop new ad hoc tools so that the new music will not sound like anything else.
The CSC organizes concerts open to the general public presenting experimental and research-related musical projects. These concerts mainly take place in the Auditorium of the Conservatory of Music “Cesare Pollini” (Padova), but our collaborations with a number of art and music organizations bring the CSC to perform in other prestigious venues around the world.
Listen to Concerto Campi Magnetici, the concert dedicated to the first 40 Years of CSC (1979-2019) and 10 Years of SaMPL (2009-2019):
See the flyer with the concert program (100MB, in Italian):
-Marco Stroppa, Traiettoria…deviata (1984, rev. 1988) per pianoforte e nastro magnetico, Produzione CSC.
-Agostino Di Scipio, Fractus (1990) per viola e suoni generati mediante computer, Produzione CSC.
-Nicola Sani, A Francesca Fortuna … Atmend nach seligem Lauf, auf nichts zu, ins Freie… (2019) per pianoforte e elettronica, prima esecuzione assoluta, Produzione SaMPL.
-Giovanni Mancuso, I Campi Magnetici (2019) 11 sogni da concerto liberamente tratti da Les Champs Magnetiques (1919) di Andrè Breton e Philippe Soupault.
On the occasion of the 40 Years of the CSC a concert with musical works realized in the 60', '70 and '80 was also organized. Listen to the history:
- Masse1 (1968), by Gruppo NPS
- Masse2 (1968), by Gruppo NPS
- Whisper out of time (1976), by James Dashow
- With the light pen (1977), by Teresa Rampazzi
- Allegro Faust (1985), by Piero Olmeda
- Una storia chimica (1987-1989), by Roberto Doati
New musical instruments use human-computer interaction systems. As an example, you can see a video of the installation PianoSpace. It is the result of the research for a new approach to the instrumental potential of the piano, up to now limited by the morphological characteristics of the keyboard-hand system. PianoSpace moves away from the piano tradition to explore the sonic potential of the piano and a kinetic control system in space through the creation of multiple Virtual Morphological Models and different Compositive Maps.
Reference:
- Vidolin, A. (1997). Musical Interpretation and signal processing. Musical Signal Processing Curtis Roads, Stephen Travis Pope, Aldo Piccialli, Giovanni De Poli (Eds), pp. 439-459, Swets & Zeitlinger B. V., Lisse, the Netherlands.
