Cutting Edge

Feature

Visualization technology creates a new dimension in watching sports

Talk 02 : Welcome to the era of watching live sports broadcasts in 3D animation.

Apr 18, 2024

Go to Introduction page

Beyond Sports started with sports analysis technology, helping teams and coaches to improve players’ performance. Now they are pivoting to change the way we consume live sports. What does the new experience look like? We talked to Sander Schouten, co-founder and CEO, and Nico Westerhof, co-founder and CTO of Beyond Sports.

  • Sander Schouten

    Co-Founder & CEO
    Beyond Sports

  • Nico Westerhof

    Co-Founder & CTO
    Beyond Sports

A Decade of Innovation in AI-Based Visualization Technology

──Beyond Sports is specialized in AI-based visualization technology. Could you tell us how the company started?

Sander：About a decade ago, while working at a tech company, I encountered a virtual reality headset. This sparked an idea to provide coaches with the player's point of view. Pitching it to the Royal Dutch Football Association led to a challenge from the national team coach. We had two weeks to create approval concepts, and despite the tight time frame, they granted us access to crucial positional data.

The concept revolved around using this data to recreate virtual players, similar to popular sports games. This idea gained traction, especially around the 2014 FIFA World Cup. Post-World Cup, an interview caught the attention of Nico from AFC Ajax Amsterdam, who shared a similar concept. Our collaboration led to the founding of Beyond Sports in 2017.

Initially we focused on player improvement and coach interaction, then we pivoted towards mass media engagement. Collaborations with major broadcasters followed, and our first game on Nickelodeon, the US children's cable TV channel, introduced Blocky, a voxel-style avatar, into a live NFL match, opening new approaches for fan engagement. Our growth has been rapid, earning recognition such as an Emmy Award in 2021 for our Nickelodeon games.

The company experienced significant growth, expanding from a two-person team to 55 employees. I oversee culture and commercial aspects, while Nico handles the technical side. Our journey is a blend of innovation, collaboration, and a commitment to redefining sports engagement.

──What made you decide to join the Sony Group?

Sander：Around the end of 2021, we needed to find a partner to help take us to the next level and realize our dreams of building unique sports consumption experiences.

We made a shortlist of three companies, and Sony was our top choice. Interestingly, three weeks later, we got a call from an accounting firm, hinting at Sony's interest. This led us to join the Sony Group.

Our main motivation was that we lacked our own tracking system and delivery mechanisms for our visualizations. Sony, with Hawk-Eye, had both of these. It seemed like a natural fit. We could use Hawk-Eye's data, upgrade and enhance it, and provide it to entities like Pulselive, a Sony-owned digital services company, and sports channel. Our next big goal is to have live matches on gaming platforms.

Another aspect is around culture. Sony and Beyond Sports share common values, such as embracing diversity and inclusion.

Beyond our technical achievements, our company's success is driven by the unique culture we've built. Our office has a special atmosphere, feeling more like a living room than a corporate space. We value diversity; with 17 nationalities and various languages among our 50-person team, we embrace different perspectives positively. It's about using those differences to improve ourselves rather than suppressing individual thinking.

──What are the core technologies of your current business?

Nico：Our core technology is split into two major pillars.

The first is data processing. This is the engine behind the scenes that makes visualization possible. It involves taking raw tracking data which is often incomplete or single-point and transforming into immersive visuals. The team working on data processing is as substantial as our visualization team. We enhance data to create complete skeletons, ensuring a lifelike representation.

The other is visualization. This is what people see on television, alternative broadcasts, and in some direct-to-consumer approaches. Our in-house teams handle everything, from 3D artists crafting the essence, environment, and lighting to a team building simulations, applications, and virtual cameras. We aim to provide a seamless and captivating viewer experience.

Nico：Both pillars work in tandem: data processing refines the tracking data we receive, and visualization brings this refined data to life for viewers. It's a complex process that involves multiple teams handling different aspects like data streaming, distribution, machine learning, mathematical analysis, and various elements of visualization, from artistic creation to automation.

In essence, these two pillars, data processing and visualization, are the backbone of our operations, driving innovation and excellence in the sports technology sector.

Pioneering Data Processing and Visualization

──Could you explain the strengths of Beyond Sports' technology?

Nico：There are two key elements that set us apart.

Firstly, the quality of player movement, driven by our advanced data processing. By combining various data sources and implementing rigorous validation and enhancement processes, we achieve a level of natural movement unmatched in the industry.

The second distinctive aspect involves our visualization layer, characterized by its flexibility and creativity. We can transform the appearance to resemble anything, like the Green Aliens of Toy Story, while preserving the authenticity of player movement. This adaptability is very unique.

Additionally, I want to highlight our focus on automation. Every virtual camera in use is fully automated, so there's no need for virtual production camera operators. This allows for a streamlined production process.

Instead of the traditional broadcast trucks and numerous camera operators, we can execute an entire broadcast seamlessly on a single computer. This innovation optimizes efficiency and elevates the virtual broadcasting experience to a new level.

──Hawk-Eye, which is also part of the Sony Group, has tracking technology. How do you collaborate with them?

Nico：The collaboration with Hawk-Eye is focused commercially on a joint vision and technologically, primarily around data.

We are data agnostic and we work with any data source that's coming in. And Hawk-Eye is one of the data sources.

While we maintain some separation on the technical side such as data validation and data processing, there are significant opportunities on the commercial front. For instance, our recent “Toy Story Funday Football” broadcast seamlessly integrated data from both the Hawk-Eye system that was under trial and the system which had already been installed by the NFL. The Hawk-Eye system is not only installed in the US, but also in Wembley Stadium in London, where the “Toy Story Funday Football” broadcast was taking place.

Nico：The collaboration has proven instrumental, particularly in instances like the installation of the Hawk-Eye system at Wembley Stadium in London for a specific game. This installation facilitated our unique broadcast, showcasing the synergy between our data processing and Hawk-Eye's optical tracking system. So it was really helpful for us.

As for Hawk-Eye's SkeleTRACK, it serves as their optical tracking technology, capturing the limb positions of players, the ball, and the referee using a variable number of cameras inside the stadium. This positional data feed from SkeleTRACK seamlessly integrates with our system, enhancing the overall virtual simulation experience.

Reaching out to the younger generation

──You mentioned “Toy Story Funday Football”Toy Story, are there any other examples?

Nico：As I mentioned, in 2023, we had significant successes with ESPN, particularly with the “Toy Story Funday Football” and "NHL Big City Greens Classic" broadcasts. Before that, we made two Nickelodeon games in 2021 and 2022.

──What are the objectives of these projects? Also, please share with us the outcomes and feedback.

Sander：In the sports entertainment industry, engaging with a younger demographic is crucial, considering that kids are shifting away from traditional television. The challenge is to capture their interest before the age of 14, as multiple studies indicate that winning them over at this stage is pivotal.

Among the traditional broadcasters, it used to be said that kids have short attention spans. But I believe how the content is curated is the issue, rather than attention span. The younger generation are raised in a world of avatars and virtual characters. They need a different approach. While they may return to live sports imagery as they grow up, engaging them now requires innovative strategies that resonate with their digital experiences.

Nico：And the results are promising. In the case of "NHL Big City Greens Classic", the median age of the viewers went down a lot. It was what we had aimed at. And there were actually more female viewers than male, which is very unique for sports.

Nico：We can provide comprehensive solutions for innovative ways to revolutionize sports content and connect with younger audiences. For the IP or rights holders, we serve as a one-stop shop.

On social media, the overwhelming response was incredibly positive. Many parents shared their struggle of trying to watch a game with their kids on traditional broadcasts, only to see them lose interest quickly. With our alternative broadcasts, for the first time, parents found a way to enjoy a match together with their kids. Whether parents choose to watch the original game on the big screen and kids on the alternative broadcast, or they chose the alternative broadcast for a joint viewing experience, it created a unique bond between parents and kids.

Paving the way for Alternative broadcasts

──Could you share your vision for the future and how virtual content generation technology will impact sports and sports entertainment?

Nico：Our long-term goal is to shift from a traditional, linear, and passive engagement with sports content to an interactive, active, and personalized experience.

While our alternative broadcast solution already resonates with younger audiences, this is only the first step. There's more to achieve. The idea is to empower individuals to control and shape their own narrative of a sports game, fostering engagement.

Sander：We want to be a pioneer of 3D animation in sports. Imagine having the ability to create your own narrative of a sports game. This opens the door to creating YouTube stars within our platform or a Sony-owned platform. By rewarding engaging narratives, perhaps through sponsorships or other means, we can revolutionize the way sports content is consumed. Instead of a single large broadcast targeting millions, we can envision thousands of channels tailored to smaller audiences, creating a more personalized and interactive experience.

Looking ahead, being part of the Sony family offers tremendous potential. We now have access to the wide variety of IP owned by the Sony Group, including movie and music. We aim to be a conduit within the Sony Group, connecting various departments and leveraging the power of sports to bring people together.

- Features articles

Related Tags

Related article

See more