Powering Live Immersive Sports Experiences with Amazing Quality

At Meta Connect, on September 23rd, Meta announced the Meta VR Glasses: a device that, as Meta stated, defines a new era for virtual reality. With a glasses-like form factor, Meta VR Glasses are a step change for enjoying immersive entertainment, which has been Tiledmedia’s focus for close to 10years. 

Needless to say that we are very excited about the new device!

While we need to wait until Spring next year for the new VR Glasses to be generally available, on October 10th, Meta launched its Immersive Sports service for Meta Quest 3 and 3S at an incredible “8K VR180” quality with an ESPN College Football match between the Texas Longhorns and the Oklahoma Sooners.

Powered by Tiledmedia under the hood, the service combines a director’s cut from several immersive cameras with a virtual “Jumbotron”: a large, virtual screen that displays the TV feed of the event in sync with the immersive feed. The virtual Jumbotron has the close-ups and replays that you would get in a “normal” TV broadcast. Users also hear the commentary that comes with that broadcast and the sound of the venue, combined into immersive, spatial audio. 

To give sports fans full context, the service further provides a live “scorebug” that’s overlaid over the immersive video. It gives score, time, and any other relevant scoreboard data such as a play clock or a counter of remaining time-outs. 

Sharp and Smooth

The end-to-end streaming of this experience is enabled by the combination of Tiledmedia’s Fabric platform and the Tiledmedia Player for Meta’s Spatial SDK.

To support the best possible immersive streaming, Meta integrated the Tiledmedia Player in its TV app as available to Quest 3 and 3S users while the cloud-hosted Tiledmedia Fabric platform performs the live 8K transcoding.

The resolution and the framerate of the Immersive Sports experience are perfect for an as-if-you-were-there, front row seat experience. It’s 8K (7680 x 4320) video at 60 frames per second, which combines incredible sharpness with butter smooth motion rendition. The service has spatial Dolby Atmos audio to match the video quality and make the experience even more lifelike.

The content looks so sharp because the resolution of the production is equivalent to about 43 pixels per degree (PPD) on average. This already looks incredibly crisp on the Quest 3 with a screen resolution of 25PPD, but it looks downright stunning on the upcoming Meta VR Glasses with a resolution of 37 PPD (!) Details about the new device, including its resolution, are here.

Further helping to preserve the fidelity of the video, it is sent to the device in the camera-native fisheye projection. Streaming the video in its native, captured projection takes away the need for intermediate projections, such as the often used but very wasteful Equirectangular Projection (ERP).

Tiledmedia Fabric prepares the content for distribution

Tiledmedia Fabric produces, encodes and packages the immersive videos for the new services, getting it ready for streaming to fans.

For the Immersive Sports service that was just launched,Fabric receives the video as an 8K fisheye feed, while it ingests the broadcast as regular 16x9 video with up to 4K resolution. Alongside the video, Fabric receives metadata feeds with live scores and game state information. There is also multi-channel audio,which may be associated with the 4K or 8K video.

Fabric then lines up the videos, the audio and data feeds so they’re in sync. Next, it overlays the TV broadcast (the virtual Jumbotron) over the fisheye feed, using a special “reverse projection” to match the fisheye video. This projection ensures that the broadcast video looks like a regular, rectangular video when it’s displayed in the user’s device. The result is a convincing virtual Jumbotron that gives the viewer the “regular” TV feed in addition to – and in sync with – the immersive feed.

Fabric also generates a scorebug graphic based on a metadata feed, and this graphic is fused into the immersive feed as well, using the same reverse projection. The scorebug is animated: the score and any clocks change in real-time, in sync with the stadium clock scoreboard.

Listening to the incoming data feed, Fabric will update the lay-out of the video when required. For instance, it automatically hides the Jumbotron and the scorebug based on the state of the game, e.g. during an a break. The platform also allows inserting slates into the feed, either automatically following game state data, or based on an operator toggle, e.g.when the game ends.

On the audio side, Fabric takes the multi-channel input and converts it into a spatial Dolby Atmos audio feed, and packages that Atmos signal with the video feed.

The resulting audiovisual feed is then encoded into an ABR ladder crafted for immersive distribution, packaged, DRM-protected if required, and then egressed to a CDN origin server.

Tiledmedia Player: tight integration for the best experience

Fans watch the Immersive Sports service in the Tiledmedia Player for Meta’s Spatial SDK that is included in Meta’s TV app for the Quest 3and 3S. Tiledmedia’s Spatial SDK player supports direct 8K fisheye playback for the best possible quality, with the least transforms and projections as explained above.

The player is tightly integrated with Meta’s Spatial SDK to optimize performance, energy use, efficiency, and perhaps most important, to enable the most faithful visual reproduction of the immersive video in Meta’s VR devices. The player and the TV App also rely on the support for Dolby Atmos in the Horizon OS, for a spectacular spatial audio experience to complement the stunning video quality.

UploadVR wrote a nice review about the series (OpenSeason on Draymond Green Shows How Well VR Docuseries Can Work) that stated: “The level of clarity is remarkable. I used my Meta Quest 3 to watch, and this was the clearest, most detailed video I have ever seen on Quest.”

Meta will continue to do live matches , so if you are in the US and have access to a Quest 3 (preferred) or 3S device, do check it out!

‍

‍