Nacho Reimat, Sergi Fernández and Mario Montagud, i2CAT researchers, presented results from the ImmersiaTV, both via oral presentation and showcasing demonstrations. Attendees were generally interested in the project’s proposals and results. During the conference, our colleagues had the opportunity to meet other professionals in the field and experience rich interactions with leaders of other research projects.

Reimat and Montagud have also been members of the Organizing Committee of the conference.

The multi-device system allows the audience to experience the same scene from several points of view. As our colleague Maria Pacheco (Lightbox) explained in an article about hybrid interactive screenwriting “we approached the script with the conceptual premise of mixing objective and subjective narratives, expanding the story using the different medium’s natural content habitat”.

The first video experience is a documentary that follows the daily life of David, a young Portuguese athlete who is pursuing his dream of becoming a successful football player. The second pilot is a live sports event.

In order to improve the experiences that we are offering, you’ll be asked to fill a short questionnaire after watching the content.

• Watch Pilot 1: a documentary
• Watch Pilot 2: a cyclocross race

We hope you enjoy them and don’t forget to tell us what you think!

The JPEG codec for image compression has been widely popular over the past 25 years, as it can reduce the file sizes of common images by a factor of ten. This compression process was developed so that images could be stored in cameras, which had limited memory, and easily shared over low-bandwidth communication networks. Yet, new applications, more efficient broadband networks and high-processing electronic devices require new solutions that can meet their needs.

Proving that new standards can be adapted to new conditions set by technological advances, on the 9th of April 2018, the JPEG committee unveiled JPEG XS, a new format for image-based compression which produces higher-quality images while at the same time requiring less energy consumption due to the reduced complexity, so that files can be transferred over high-speed networks with low delays. The JPEG XS standard compresses still images by no more than a factor of six; however, it is meant neither to compete nor to replace the popular JPEG format. Conversely, it aims to complement it by providing a suitable alternative for new applications on virtual and augmented reality, space imagery, autonomous driving, and professional editing. Although the compressed files end up being larger in terms of size, they are aimed to be directly consumed onto electronic devices rather than being stored, by exploiting the instantaneous delivery that can be achieved through new, high-speed broadband networks, such as 5G.

According to Professor Touradj Ebrahimi, an active member of the ImmersiaTV project who is leading the JPEG committee, “For the first time in the history of image coding, we are compressing less in order to better preserve quality, and we are making the process faster while using less energy”. He says that “we want to be smarter in how we do things. The idea is to use fewer resources, and use them more wisely. This is a real paradigm shift”. Regarding the visual quality of the encoded images, “Compressing the images with JPEG XS doesn’t compromise quality at all: even experts can’t tell the difference between an original and a compressed picture or movie”.

Applications that require faster transmission of higher quality images will benefit from such a standard. The near-lossless compression results in fewer artefacts in the encoded image, which in turn leads to high accuracy in the recognition of objects, for instance, while fast delivery is boosted by the low-complexity encoding implementation. Autonomous driving and manipulation of unmanned aerial vehicles (i.e., drones), are just a few examples of use cases. The minimum energy consumption, along with high-quality image acquisition, has also attracted the attention of the European Space Agency (ESA), among others. Smartphones are expected to support this standard in the near future, without replacing the legacy JPEG format that is employed for image capturing and storage, for immediate and high-quality media consumption.

Following the tradition of JPEG in standard releases, this format is also open source, which is a significant advantage for its adoption and integration in a wide range of applications. “In the immediate future, JPEG XS will be put to use in professional applications like movie editing, space imagery and professional-grade cameras. Consumer electronics will come next, including self-driving cars, virtual reality, augmented reality, and wireless connections between multimedia devices and TV monitors or projectors”, says Ebrahimi. “To be able to use JPEG XS, consumers will need to own the next generation of devices. In terms of software, they will probably just need to run an update, as they do from time to time on their computers and smartphones anyway”.

Original article:
https://actu.epfl.ch/news/a-new-jpeg-format-for-virtual-reality-drones-and-s/

The representatives of the project were Gregg Young from VRT (Belgium), Simon Delaere from IMEC (Belgium) and Juan Antonio Núñez from i2cat (Spain). Young, Delaere and Núñez agree that the project attracted visitors interest, even though their stand was in a less busy zone than where the big industries were. The majority of the people that approached the stage had a direct interest in immersive experiences and, generally, were impressed with the solution, since they understood the objective of the project and its value.

This experience has been an excellent opportunity to assess the project effort in the US broadcast market, which is considerably different from the European one. The researchers could also see that the evolution of the project and the strategic decisions made during these years are aligned with the market demands and with the most innovative projects presented at the fair.

Overall, it was a perfect occasion to figure out, from a strategic point of view, if it’s possible to transform a research project into a profitable product.

From the 9th to the 12th of April, NAB invites Belgian public broadcaster VRT to exhibit the results of two research projects, funded by the European Commission and Flanders Innovation & Entrepreneurship, in the Futures Park. Focusing on co-creation, interaction, and immersion, these projects look into new ways for utilizing novel technologies and formats to engage users in the creation and experience of events and media.

The two projects, called MOS2S and ImmersiaTV, can be found at the Futures Park in the North Hall, at booths N1539FP and N2134FP.

Extending live event experiences with user-generated content

MOS2S is aimed at enriching live event fan experiences. The project is driven by an international and industry-led consortium of 15 partners, led by TNO and spanning 4 countries (The Netherlands, Belgium, The Republic of Korea and Turkey). In cooperation with project partner Kiswe, VRT’s focus is to enable event audiences to co-create the experience with user-generated content, interactivity and live interviews. The project builds upon “Wall of Moments”, an interactive mobile app developed by VRT, that enables event visitors to upload photos and videos of moments that professional cameras cannot always capture. By sharing a live stream of photos and videos, event organizers are able to interact with the audience and capture the unique experience and atmosphere of their event, through the eyes of the visitors, for their friends present or at home to enjoy. The Wall of Moments is also demonstrated at NAB.

Innovative and immersive broadcast content

ImmersiaTV extends traditional tv broadcast content with omnidirectional video experiences, synchronously published on tablet and head-mounted display. Viewers are literally able to jump into the action during a live broadcast event, such as sports or entertainment, and experience unique places as “the best seat in the house”. ImmersiaTV delivers a full production and distribution pipeline that can be integrated in traditional broadcast environments.

As an aspiring screenwriter and filmmaker myself, having the opportunity to conceive a new narrative experience with the Immersia TV tools was nothing short of exciting, from an artistic standpoint. Being a fast paced, highly creative production house, this is the sort of project that excites us at Lightbox.

In fact, the whole premise and basic conceptual idea for what would be the script for the offline third pilot was something that came to me when we were first working on the first one. Not only were we breaking in new methodology to produce content within this new paradigm, but we also started bouncing off ideas of what this new combination of mediums could possibly allow for.

During IBC, as a part of documents written to make a case for the potential implications of Immersia TV, one in particular resonated strongly with us: if a 2D image, or directive video, on a television could be seen as something objective, could we expand the experience by providing the user with the same narrative, only made subjective? A clear simple example was used, in a very Soviet Cinema School like fashion: if we saw on the TV a child being scolded by his mother for a broken vase, in that isolated event, we, objectively speaking, process the information that he or she broke the vase and now, the mother is livid with anger. This would be the scene shown on the TV, through 2D directive images – objective narrative information. On the upside, what if we took that very same scene, but on a 360º omnidirectional image? What would be the difference?

With a directive image, your eyes are directed towards a specific thing the scene must (or must not) convey. In omnidirectional, the entire area of a certain scene is always open and action is all transparent. With this in mind, let’s take the next example: the same scene, mother scolding a child. Only this time, when we’re in the 360º image, if we look hard enough, maybe we can see the younger brother hiding behind a cupboard with a naughty smile on his face. This not only changes our interpretation of the action, but it also expands the narrative by providing us with we could say is subjective narrative information.

 This is the basis from which the script was approached, with the conceptual premise of mixing objective and subjective narratives, expanding the story by means of the different medium’s natural content habitat.

When the script work started, the main issue was to build a synergy between different platforms in terms of narrative. This meant writing cohesively to the strengths of each medium. In order to meet such condition a genre of story had to be chosen having in mind the main conceptual challenge of the pilot. A Whodunnit detective story felt like an appropriate sandbox to play around in, always having in mind the general story experience we wish to create and share with users. This also means that the screenwriting process becomes something akin to looking at three places at once, writing with both hands, and juggling a football, all at the same time. Upon deciding on the overall narrative which to explore in such a format, a decision must be made regarding which medium is better suited to expand the narrative experience.

Essentially, I took my own screenwriting process and adapted it to the immersia paradigm. First, I drafted an outline. After having a clear idea of the structure, a common 2D screenplay is written, just like any teleplay or film script. At this point, only the TV portion of the project is produced, script wise. This serves as a blueprint for the remaining two platforms: head-mounted display (HMD) and second screen (tablet).

On the HMD, we can take advantage of the 360º space and expand the narrative, using the TV content as a starting point. On the second-screen, we focus on interactivity, by tapping into the tactile dimension of the tablet. But what does this mean in terms of the actual script and its creation process?

The first step is to create a traditional screenplay that will be the base from which we work. The second however, is where it gets interesting: we needed a way to view all platforms, their content and interactions simultaneously, in order to have a general view of the entire narrative arc and experience. This resulted in a three column grid-like document that allowed that.

Vertically, I track the time of action and it’s chronological order: this happens, then this and that,

and so on  and so forth… Horizontally, I establish the connection between platforms:

The main idea behind this organization is to travel between columns, from content to content. Things could get very complicated when it came to pre-production and planning, despite all the experience we had in making pilot 01. However, there is a big difference between the two pilots: in the first one, content was continuously synchronized (that was its basic conceptual premise) across all platforms. Pilot 03 is a step up in that regard: we designed an experience that flows between devices not requiring content to be synchronized. This seemingly small change opens up new possibilities for the production process: from prep to post, there can be a separate planning  of what to shoot – one for directive, another for 360º – meaning that while in production we can film by type of footage we want to capture. As a final result, post-production, in particular the editing, becomes simpler in its structure and work methodology on the directive side, while becoming much more complex when it comes to 360º and interactivity.

The process of creating a script from which it would be possible to build a narrative preview of the interaction and synergy between the different devices, was an entirely new one. As an almost blank canvas of narrative possibilities such as this one, mixing 2D with VR and interactive touch, would most likely benefit from high profile industry level screenwriters and filmmakers, from all the around the world, to experiment with the Immersia TV paradigm. Only by experimentation, failure, repetition, failure, repetition, success – the methodology of scientists and artists – can we truly bring forth something new and unique.

This was joint collaboration between Poznań Supercomputing and Networking Center, Poznań Opera House and Colour Pictures Paweł Zabel, directed by Pawel Bondara. The traditional video content was recorded and transmitted using twelve 4K and HD cameras connected to OB-Van, where the live production was done. The traditional 2D recordings could be watched live at the OperaVision platform and are still accessible here.

Additionally, we captured 8K video clips and 360 clips for Immersia TV project. The content is still in production but here’s a sneak-peak!

The recording will take place on February 10th during the general rehearsal of the opera, and it is an initiative by the H2020 European project Immersive Accessibility (ImAc), which has joined forces with ImmersiaTV to record and produce the omnidirectional piece.

The result will be an interactive and immersive product in which the viewer will be able to choose between the realization of classical television and alternative views from different spaces, both on and off the stage.

The objective of this experience is also to test accessibility in immersive environments, the main objective of the ImAc project.

ImAc and ImmersiaTV

The ImAc project, led by i2CAT, began in October 2017 and explores new methods of implementing accessibility services (subtitles, audio description, audio subtitles, sign language and accessible interfaces) in immersive 360º video environments.

ImmersiaTV, proposes a 360 degree virtual reality multi-device solution that covers the entire value chain of the audiovisual sector. Researchers at ImAc will add a layer of accessibility services to the format generated by ImmersiaTV to facilitate and improve the experience for people with specific needs.

Text and pictures: i2CAT

A straightforward way to produce ambisonic sound is using a soundfield microphone. It consists of high quality microphone capsules arranged, with their polar pattern taken into account, in angles and directions that allow covering the whole 360° sound sphere. Their frequency response and sensitivity should also be carefully matched. What is more, for good directionality of high frequencies, due to spatial resolution of sound waves, they should be packed really close to each other – ideally they would be placed in the same point.

Most ambisonic microphones on the market are made of four small capsules, with cardioid pickup patterns, arranged in a form of a tetrahedron. Four channels on the direct output represent ambisonic sound in so called A-format. Usually manufacturers provide software tools dedicated to their microphones to convert it to standard B-format ambisonics track. Some popular models and companies that produce ambisonic microphones are: Sennheiser AMBEO, Soundfield SPS200 and ST450, Rode Videomic Soundfield.

Soundfield microphones are a perfect solution for recording ambient noise – sounds of the street, nature, etc. However, when recording speech, we can’t really expect the highest quality – mostly due to the distance to the microphone. It is much better to add for certain sound sources, like speaking actors, traditional e.g. Lavalier microphones and then adjust levels and mix them with ambisonic background. First issue here is that we need to control delays. Otherwise, an unwanted short echo effect or sound phase shift could appear, as soundfield microphones still records the same source but from different distances.

To mix traditional recording into ambisonic format, we need to transform it – simulate what soundfield microphones would ‘hear’ from the sound source. In this conversion we don’t lose quality, but we need the right tools. Basic equipment for such processing is in general called DAW – Digital Audio Workstation. It is a common name for software or devices for recording sound, editing it and manipulating (filters, effects, loops, etc.) to produce digital audio content. The DAWs market provides whole range of solutions from cheap or free applications running on a commodity PC to complex hardware stations. Modern DAWs are easily extended using software plugins that provide additional functionalities. For ambisonic sound production plugins worth checking are e.g ambiX, atkToolkit or O3A Core.

Encoding of traditional sound track into an ambisonic format requires a relative position of sound source and virtual microphone. Additionally, it can be also adjusted by the attenuation of objects between camera and actor. It is not a problem for static sound sources, but it becomes an issue with moving actors. Usually, it requires an experienced operator that ‘animates’ the sound (with 6 degrees of freedom – rotation also makes a difference!). With more actors or live productions, it’s usually better and more convenient to use an automatic tracking systems.

In fact, with this approach at production side we create virtual sound environment consisting of audio objects. The final format depends on the intended use. It all can be mixed as an ambisonic sound, but for highly interactive scenarios the final mixing can be left for the user application.

Authors: Mikołaj Węgrzynowski (PSNC), Eryk Skotarczak (PSNC)
Photo: Eryk Skotarczak (PSNC)

Cyclocross is a very popular sport in Belgium, and SOUDAL Cyclocross Leuven is one of the main events taking place yearly. The ImmersiaTV team wanted to take the emotion wherever you are thanks to omnidirectional, immersive content, so last January 7th we placed 360 video cameras around the field and shot the race.

Users of the content were able to watch the race from different points and experience the feeling of standing in the crowd, even seeing it from locations where the audience is not allowed to be. Take a look at how we did it!