Paul Trilsbeek, Hennie Brugman, Han Sloetjes & Peter Wittenburg, Max-Planck-Institute for Psycholinguistics

Ontogenesis of Multimedia Supporting Tools

At the Max Planck Institute for Psycholinguistics, it became obvious at an early stage that multimedia recordings would play an increasingly important role in various sub-disciplines of linguistics. With MediaTagger, a first multimedia annotation tool was already presented in 1994, based on a careful database design for multimodal annotations. However, it turned out very quickly in the daily research work that this tool was too limited in various dimensions.

Based on the experience with several concrete projects involving multimodal annotation, such as in gesture analysis, and on the theoretical insights gained from the international discussions about flexible and possibly more generic annotation formats (Bird& Liberman, Brugman& Wittenburg) a new approach was taken with the development of ELAN and the corresponding ELAN annotation format (EAF). During the past years the EAF schema has demonstrated its representational power, i.e., it has time linking and symbolic references mechanisms that are widely compliant with suggestions such as Annotation Graph, it can incorporate existing annotations created by tools such as Shoebox, CLAN and Transcriber and it allowed us to include complex annotations such as those created for the Dutch National Spoken Corpus, for example. With ELAN we provide a professional tool that allows the creation, extensions and analysis of such annotation structures. Painful experiences with the first chosen media frameworks lead us to a complete redesign of the ELAN software to meet the accuracy and stability requirements.

During the last years we recognized that an increasing number of researchers integrate their annotated media files into an archive and consequently would like to utilize this archival content via the web. Therefore, it was necessary to developed ANNEX, which can be seen as an "ELAN version" that operates via the web, offering similar views on annotated media. Of course, media has to be delivered in a web-streamable format, such that when selecting an annotated fragment for example, only the relevant media fragment is downloaded and shown immediately. Due to the inherent inaccuracy of media-streaming via the web, the first ANNEX version does not allow to create annotations. However, working via the web is getting so attractive that users already asked for an extension such that annotations can be modified and even created, in spite of the dangers of creating timing errors.

In the domain of lexica we could build on much expertise and analysis work in the field with respect to generic lexical structures. Also here we were confronted with the need to integrate media objects in a simple way and access them via the web. The flexible LMF format allows the inclusion of images, sound and video fragments or pointers to fragments. In lexica however, the media playback requirements are less strict, since normally media objects are meant to illustrate a lexical entry or attribute.

After 12 years of development and lots of struggling with timing accuracy and media streaming, we can now claim that we can offer a set of mature and accurate tools for the major linguistic data types, annotations and lexica.