Cutting Edge

Special feature

Talk 01 : The Infinite Possibilities of the Tracking System "SkeleTRACK" - Hawk-Eye Brings "Sports Revolution"

Apr 8, 2024

Go to Introduction page

Established in 2001 by British entrepreneur Paul Hawkins, Hawk-Eye stands as a pioneer in the sports technology field. In 2011, the company achieved a milestone when it was acquired by Sony Europe, solidifying its position as a leader in the industry.

We spoke with Alison Marr and James Sharam from Hawk-Eye about its technologies and how they are changing sports and sport entertainment.

  • Alison Marr

    Head of Tracking Products
    Hawk-Eye Innovations Ltd.

  • James Sharam

    Head of Computer Vision Engineering
    Hawk-Eye Innovations Ltd.

The founder was also a cricket athlete

Hawk-Eye was founded in 2001 by Dr. Paul Hawkins O.B.E. The concept had initially been ideated through an “Innovation Amnesty” exercise at Roke Manor Research in 1999, where Dr.Hawkins had been working at the time. Hawkins, having been a cricket athlete himself, began testing the application of trajectory tracking in cricket, and it quickly expanded to tennis, football and the wide array of sports, developing video and tracking technologies that are now both industry-leading and industry-standard on a global scale. Hawk-Eye was then acquired by Sony in 2011, along with sister-company Pulselive, and has continued to grow successfully ever since.

──What is your current position and role?

James：I am the head of Computer Vision Engineering at Hawk-Eye and my responsibility is to build end to end computer vision solutions for tracking purposes. That will be anything from capturing cameras to running AI models, to generating data feeds and other outputs for the customers to consume and we manage that whole ecosystem of software from capture to delivery.

Alison：I am the head of tracking products at Hawk-Eye. As a part of the product team, I look after all of the tracking products that we have across all the sports. Working closely with James on the engineering side to develop the products and push them through to market.

Revolutionizing Sports Analytics: The SkeleTRACK Innovation

──One of the recent innovations from Hawk-Eye is SkeleTRACK, could you tell us what this product is?

James：SkeleTRACK is a product which we first released in 2019. It is primarily a system for real-time tracking of people and objects such as balls and bats. Although its capabilities extend to tracking other elements, such as monitoring the pitch to determine camera locations and tracking additional markers on the field, its main purpose is to generate real-time skeletal data for athletes on the field, teams, broadcasters, fans, and more.

Currently, the system monitors 29 key skeletal points on an individual's body, including the shoulders, hips, ankles, and wrists.

Beyond raw tracking, SkeleTRACK can generate sport-specific events as it detects kicks, throws, or catches. It provides detailed, frame-by-frame information about the location and identity of every individual on the field, allowing the derivation of significant events that mark the progress of a game.

James：Operationally, SkeleTRACK involves the placement of several cameras around the field of play. These cameras may be situated in the gantry of a stadium/arena or on tripods along the pitch perimeter, depending on the specific setup. The captured images from different perspectives undergo analysis through various AI and machine learning models. This analysis enables the system to determine individuals' positions in each frame and calibrate the images accordingly.

And then, the system processes the gathered views and detections, turning them into a 3D representation using the calibration data. This comprehensive approach ensures accurate and dynamic tracking capabilities for a variety of applications.

Alison：We also track other objects on top of just the skeleton. For baseball, we track baseball bats and for hockey, we track their sticks, things like that. We work closely with our league partners to determine what data is valuable to them to create downstream benefits.

──What sports is SkeleTRACK used for?

Alison：SkeleTRACK can be used for quite a few different sports, but at the moment it is mainly used for football, baseball, basketball, tennis and ice hockey.

In baseball, data is crucial to teams, players, performance analysts and front office staff.

And in football, we also supply data, but there's an officiating component as well. We have a Semi-Automated Offside Technology that utilizes SkeleTRACK's high-precision and real-time skeletal and ball tracking technology. It's something we've been developing over the past few years in collaboration with major football federations and competitions.

──How does SkeleTRACK differentiate itself?

James：SkeleTRACK is unique for a few reasons, it’s market-leading accuracy, we have to supply the best technology in the market due to the officiating use cases that we support. The granularity of the data available is what makes our system standout. No other competitor covers 29 points on the body in real time, many tracking systems can only cope with Centre of Mass data from optical-based tracking systems, but our system unlocks many use cases for biomechanics/performance analysis. Our ability to do this at low latency, as well as a managed service approach that ensures highest possible quality through scrubbing, and at scale, makes it truly unique in-market.

We’ve also built the system based on our breadth of experience across different sports and use cases to create a best-in-class model. We can deploy SkeleTRACK in a variety of environments, this is because of our understanding and ability to address the challenges of physical installments in arena versus in stadium deployments, including things like the ability to run our systems in different lighting and weather conditions. The system can also run on cloud as well as on premise as the sport or client requires. The use cases of the data are also endless, from fueling officiating tools to performance analytics, athletic welfare, to creating sources of entertainment such as data over video, virtual recreations, and augmented reality (AR) experiences, using SkeleTRACK opens up a whole new world for our clients.

──Could you elaborate on the potential for integration with other Sony Group technologies, such as image sensors and video technology?

James：We have already established some partnerships with Sony on specific projects.

For instance, there is a collaboration between Manchester City Football Club and Sony, where they utilize date from SkeleTRACK, which was installed at the stadium. That's one area where integration is already happening.

There are numerous possibilities for collaboration in that realm. We're exploring areas that could benefit from both computer vision and broadcast technology, which could yield powerful results. Accurate and reliable skeletal person tracking and identification have practical applications in various fields.

The potential extends to diverse use cases with Sony, including performance analysis, athlete welfare, and more.

Alison：We've conducted a few small proof of concept (PoC) projects with different research groups within Sony in the past. I hope we can progress them into concrete developments in the near future.

Inside the Game: SkeleTRACK’s Pioneering Role in Baseball and Football

──You’ve mentioned SkeleTRACK’s current main use cases are in baseball and football, could you provide more details on your work with teams?

James：Generally, our primary use cases in football for the data involve officiating. Knowing the players’ positions, ball location, and game rules allows us to automate certain officiating decisions, like detecting when the ball goes out of play.

Another key application is virtual recreation. By tracking players’ movements, we can recreate events in a virtual world, offering different perspectives, such as a first-person view, providing insights into players’ perspectives and actions on the pitch.

The tracking and calibration technology also aids in overlaying data onto broadcast graphics. Identifying players, their positions, and camera calibration allows us to overlay information on live video broadcasts.

James：From a data science perspective, there are implications for understanding player movements, game strategies, and formations, providing valuable insights for coaching and team training.

Biomechanical data, particularly in baseball, is used for coaching, helping improve players' techniques by analyzing their movements and comparing them to others.

Alison：We provide ball tracking data, bat tracking data, and comprehensive tool tracking data to MLB and certain clubs in Japan, such as Tokyo Yakult Swallows. This is a significant aspect from their perspective. Specifically, we're in a testing phase, providing real-time data for officiating and live broadcasts, particularly for measuring the strike zone and validating umpire calls.

For Serie A, we serve as their main player tracking data provider, offering ball tracking, player center of mass, and skeletal data. We also provide Semi-Automated Offside Technology.

──What kind of problems do you think SkeleTRACK is solving? What changes have you noticed before and after the adoption of technology?"

Alison：I would say officiating is a significant aspect. Technology in officiating provides players and fans with confidence that correct decisions are being made, ensuring fairness for all teams.

From a coaching perspective, having the extra data can be valuable. For instance, in baseball, monitoring a pitcher's fatigue or detecting signs of injury can be crucial information.

James：It also contributes to broadcast-related aspects. Live statistics from the data provide interesting insights for fans, such as the height of a hit ball or the speed of a serve in tennis.

Additionally, technology speeds up processes like offside decisions. Automating such solutions makes decision-making quicker, enhancing the overall flow of the game. Fans spend less time waiting for decisions.

Alison：We've conducted calculations with our Semi-Automated Offside Technology. Before its implementation, offside decisions were mostly made manually through VAR (Video Assistant Referee) in football. We compared the time taken for manual decisions versus decisions with the semi-automatic technology. The time was significantly faster from over 1 minute down to 15 seconds with the semi-automatic technology, providing proof that it has expedited VAR decisions, which could feel slow at times.

The Evolution of Sports Technology and Beyond

──Do you have any new projects involving SkeleTRACK? Could you share the details with us?

Alison：One of the most exciting projects is our collaboration with the NBA.

We recently became their official player tracking data provider, coinciding with the start of the basketball season in 2023. It's been a thrilling project, presenting unique challenges, especially with basketball's dynamics and the presence of very tall players.

Initially, we began by providing real-time data, featuring the 29 key skeletal points, as James mentioned. However, the collaboration extends beyond that. We are actively working on automated officiating solutions tailored for basketball, adding an innovative dimension to the project.

This ongoing collaboration with the NBA will also incorporate HawkVISION (technology for visualizing play in CG), particularly focusing on the virtual replay aspect in the future.

It's an exciting venture that holds great potential.

──How about product enhancement?

James：Increasing the number of skeletal points is definitely on our radar. We've already progressed from 18 points in our initial iteration to the current 29.

Apart from that, we're exploring various ways to enhance our systems. One is about the ways of deploying our technology. We are looking at the ability to run processes on more edge computing devices, making our infrastructure more flexible and reducing the need for extensive cabling.

We're also considering deploying components in different locations, such as on-premises, in the cloud, or directly on cameras.

It's an interesting area of development for us.

──Could you share your thoughts on the future of the sports and sports entertainment industry, especially when incorporating technologies like SkeleTRACK and HawkVISION? What possibilities do you foresee?

James：I believe the technology applied in sports is crucial for fostering fairness, as Alison mentioned earlier.

Additionally, innovations in virtual recreation, particularly through partnerships like Beyond Sports, have the potential to attract younger audiences by offering unique perspectives and insights into sports. This can bring a different way of enjoying sports, as it provides viewers with multiple angles, deeper investigations, and even playful themes like the recent collaborations with Beyond Sports, where our tracking data was used to create distinct visual experiences, such as the NHL's 'Big City Greens' and NFL's 'Toy Story Funday Football'.

Alison：SkeleTRACK's tracking data opens up a whole new world of real-time information during in-game play. While skeletal information has been available in labs before, having it in the actual game environment is a game-changer. This information allows for intriguing possibilities, such as predicting injuries by detecting subtle changes in an athlete's running pattern, enabling timely substitutions.

Additionally, we can now delve into specifics, like how hard someone kicks the ball with their left or right foot. It's a treasure trove of insights that was previously unavailable and opens up exciting new avenues in sports analytics.

- Features articles

Related Tags

Related article

See more