Imagine live streaming a music concert in high-quality VR video: you turn to your favourite guitarist playing an awesome solo, hearing every bit of reverb coming from the guitar, and you then turn to hear the overwhelming cheers from the crowd after the song ends…
The three most important pillars of compelling media experiences like the one described above are the visual action, the chance to interact with friends and strangers, and the sounds of music, cheering and suspense around you.
Tiledmedia’s software is a vital component of all three pillars. Firstly, our tiled-based streaming solutions – ClearVR and Mosaic Multiview – optimize the distribution of the visual action. Secondly, our Tiledmedia Player SDK supports social synchronization, making it possible for users attending an event to interact with each other within a virtual world. And now, we are very happy to announce that our software can now hold the weight that rests on the third pillar, with support for Spatial Audio added to our software stack.
In this blog, we will cut through the noise and explain the ins and outs of Spatial Audio, as well as the added benefits.
Fine-tuning the definitions
Let’s first define what Spatial Audio entails. Spatial Audio is a technology that lets users hear sound coming from degrees around them. Spatial Audio involves sound direction, distance, externalisation, and presence, in order to give the user an immersive audio experience.
In the context of watching immersive content, Spatial Audio adjusts the direction and source of sound based on the orientation of the viewer. This is applied to VR headsets, mobile phones, tablets, and PCs. These devices have built-in gyroscopes (directional sensors) that determine the user’s position and viewing direction. When playing Spatial Audio on a mobile device, this positional information is then sent to the headphones which will emit immersive sound.
From Mono to Stereo to Multichannel to Ambisonics
Let’s turn our heads towards the technical side of this topic. An audio stream that accompanies a video stream used to have either one (mono) or two (stereo) channels. Or, there could be many more, as in a surround sound system.
To realize immersive sound in a headset something more is required than just immersive audio: dynamic spatialisation, or binauralisation. The art of moving chaning the sound in such a way that it seems to be stationary in space, even as you move your head. A pupoluar format for representing spatial audio in headsets is called “Ambisonics”, a full-sphere surround sound format. Ambisonics represent sound sources in any direction, in the horizontal plane as well as above and below the listener. Ambisonics can be matched to head orientation and be decoded as binaural audio, which is headphones. Overall, Ambisonic audio has proven to be an effective way to present spatial audio in Virtual Reality applications, like YouTube 360 Video.
Ambisonics can be recorded with a various number of audio channels, first order Ambisonics has four channels, second order Ambisonics has nine channels and third order Ambisonics has sixteen channels. The higher the order, the better the audio resolution, which enhances the sense of immersion that comes from the audio.
To capture Spatial Audio in the form of Ambisonics, on-site production teams have to set up microphones that can record the multiple channels, but luckily the hardware is already widely available, and production crews are becoming more and more familiar with it.
Besides this, the biggest players in media and music are also moving towards mass-adoption of Spatial Audio. For example, Apple is enabling Spatial Audio for its consumer devices and headphones starting with iOS 15. This illustrates a general move towards immersive sound, which means streaming media companies will have to keep up to stay in the game.
Tiledmedia’s ‘Master of Soundwaves’ Sergio working on Spatial Audio Support
Adding the WOW-Factor
As mentioned above, our development team recently upgraded Tiledmedia’s software to support up to 3rd order Ambisonics for all kinds of video, VR 180- and 360-degree video as well as traditional 16:9 video. Tiledmedia is already known for being able to handle almost any number and type of video streams out there, and now we can safely say the same for audio.
Streaming platforms and broadcasters can now get stunning visuals, user-interactivity, and cinema-like audio, all combined within a single software package: the Tiledmedia Player SDK. No longer do streaming apps need separate software for video and audio, reducing cost and preventing A/V bugs.
This blog started with an example of a practical application of Spatial Audio. Live streaming a music concert is a great use case for Spatial Audio. Users can focus on specific band members and the sound of their instruments, or focus on the roars coming from the crowd. Tiledmedia’s Spatial Audio support, together with ClearVR or Mosaic Multiview (or both) streaming solutions, gives users that front-row feeling that participants in the ‘Streaming Wars’ are looking for.
Listen up!
Hopefully this explanation has pricked up your ears! if you want to experience the immersion that spatial audio and ClearVR can bring, you can download the Tiledmedia Player (on iOS, Android, Quest) and play the Spatial Audio clip with your headphones on or using the speakers on a VR headset. If you have a suitable use case for implementing Spatial Audio, please reach out to us, we are happy to discuss a solution fit. We can also encode your own content so you can try it out!
Author