Data, technology and…opera?

Yes, believe it or not, researchers in Insight are using technology to increase the accessibility of opera as an art form. The Traction project aims to use technology for the co-creation and consumption of opera content. The following is extracted from a piece written by collaborators on the project, Anderson Simiscuka, Mohammed Amine Togou, Noel O’Connor and Gabriel-Miro Muntean.

Traction is happening at a time in which more and more people are subscribing to video streaming platforms and watching content online, due to the Coronavirus pandemic. BBC2 reported that viewership on streaming services are 71% higher in 2020 in comparison to the previous year in the United Kingdom. It is fair to believe that the same has happened in the rest of the world with people spending more time at home.

Even though revenue for platforms such as YouTube and Netflix are on the rise, the increase in online traffic presents challenges in relation to network issues. Both Google’s YouTube and Netflix were forced to limit streaming quality in order to ease pressure on network providers this year, especially during the initial lockdowns. This decrease in video quality was only possible due to an interesting feature of these platforms: adaptive streaming.

Adaptive streaming is the ability of media players to select a video stream that suits the current network condition of the user. This is possible due to the conversion of videos into various resolutions, once they have been uploaded to YouTube, for example. That is how Google and Netflix were able to select lower resolutions for videos for users (as they were already available in different encodings) and decrease the burden on networks, as streaming of lower resolution videos consumes less data than high definition ones.

Adaptive streaming and the technology that enables it, MPEG-DASH, are very important to TRACTION. Why? Because TRACTION requires the development of collaborative multimedia players which will be used by communities in various locations, including areas with limited Internet bandwidth. It is also important that content from multiple locations can be processed and, when needed, merged, without latency or delays.

Multi-source multimedia players must support, for instance, the streaming of multiple pre-recorded recordings and live content from artists playing different instruments, merging the videos into one single experience, even when content is located in different locations. Other video elements that can be played simultaneously with a video stream include user feedback in video, commentators, and sign language interpreters.

Another important feature of TRACTION is support for immersive experiences. These include videos recorded in 360° and virtual reality (VR) applications. Content delivery of 360° videos brings additional challenges to research and development, as files are large and VR headsets require very little streaming latency to prevent user dizziness.

The proposal of adaptation schemes specifically designed for improving the delivery of immersive video is one of the research directions within the project. Image quality, especially in the context of opera content, can be adjusted for improved user experiences.

360° VR videos and the underlying 3D geometry can be divided into spatially partitioned segments/tiles in the 3D space, and be adapted with more or less priority, according to the regions the users are more likely to look at. Colour can also be improved: for instance, 360° content recorded in theatres also films the audience, a region that is normally dark where the quality can be decreased as users are not interested in seeing it. The overall video size is then decreased, improving performance.

The design of applications and solutions that support a variety of devices requires the development of novel adaptive algorithms and schemes based on device, user, and network requirements. TRACTION is a great opportunity for creating and implementing these algorithms and schemes, which must allow multiple concurrent users located in areas with limited Internet bandwidth and with a variety of devices to access and produce content at higher quality, even in constrained environments, during these challenging times.

Acknowledgements: This work was supported by the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 870610 for the TRACTION project. The support of the Science Foundation Ireland (SFI) Research Centres Programme Grant Numbers 12/RC/2289\_P2 (Insight) and 16/SP/3804 (ENABLE) is also acknowledged.