Cutting Edge
Hawk-Eye
Making Sports More Exciting Through Visualization Technology
Jan 20, 2021
In the professional sports world, there are always moments that unfold too fast or are too complicated to be tracked by the human eye. Using visualization technology, Hawk-Eye supports officials who have to make judgements in these situations. Hawk-Eye’s mission is to create technology that make sports safer, fairer, more engaging and better informed.
Profile
-
Taro Yamamoto
Sports Business Dept. Consumer &
Professional Business Sector
Sony Imaging Products & Solutions;
Sports Entertainment Business Dept.
Creative Division Sony PCL Inc. -
Mathew Richards
Technical Director
Hawk-Eye Innovations -
Edward Hawke
Product Director
Hawk-Eye Innovations
Visualizing moments that officials are unable to judge
──First of all, please tell us about the technology of Hawk-Eye and the fields in which it’s used.
Edward Hawke：The two main technologies that Hawk-Eye provides are ball tracking and Synchronized Multi-Angle Replay Technology (SMART) video replay. Ball tracking technology has been integrated into various services such as Electronic Line Calling (ELC) as well as Goal Line Technology (GLT), and SMART video replay has been adopted by football Video Assistant Referees (VARs) and rugby Television Match Officials (TMOs).
──What is the story behind the development of Hawk-Eye services?
Taro Yamamoto：Hawk-Eye Innovations Ltd. was established in 2001 by Paul Hawkins, the founder of Hawk-Eye, formulated the idea of ball tracking based on missile tracking technology. Initially the technology was developed to make cricket more enjoyable for the viewers, but quickly expanded to cover a variety of other ball sports, such as tennis, football, badminton, and volleyball. This desire to make sports more enjoyable for viewers is still present in the current business. Hawk-Eye really entered the global spotlight in 2006 following the introduction of the new challenge rule at international tennis competitions. This rule allows each player to question the umpire’s judgement up to three times in a single set. If a challenge is called, the trajectory of the ball and where it landed is analyzed using video footage taken by multiple cameras installed around the court, and within mere seconds, it is replayed on screen at the venue through computer graphics (CG). This technology can accurately analyze the trajectory of high-speed balls in tennis, which of course makes matches fairer, but it also gives rise to scenarios which critically influence the way games unfold.
Mathew Richards：Ball tracking technology has two main components - optical 2D vision processing (finding the center of the ball) and 3D triangulation (modelling the ball’s flight over time). Typically, this utilizes between 8 and 12 cameras running at frame rates of up to 340 frames per second. The data captured is fed into a central control system that can deliver a variety of real-time services, including officiating solutions, broadcast enhancements and data streams.
──How has the use of SMART technology spread?
Taro：SMART video replay technology became well-known as a tool for football VARs. By the time it was to be introduced, Hawk-Eye had created a multi view solution that would allow officials to check synchronized footage from various different angles taken by multiple cameras.
Mathew：To enable a video platform to simultaneously render and supply several different video streams, we have developed a technology to encode video at up to 340 frames per second in real-time with minimal image degradation, and to encode multiple resolutions so that the output device can select the optimal proxy video size on a per-frame basis. As a result, officials, medical staff, and broadcasters can access scenes of the footage from any angle they want at the same time.
Edward：The parts of the footage that officials, coaches, and medical staff want to check differ, and as it used to only be possible to send the same footage to everyone, the selections were left up to the operator. SMART Replay solves this problem without the need for further equipment or operators.
Hawk-Eye’s strengths,
which have brought about great innovation
──What factors contributed to the widespread use of Hawk-Eye and its successful development in the sports industry?
Mathew：Hawk-Eye has over 20 years of experience in delivering ball and player tracking services, and one of our key strengths is our ability to constantly innovate and refine our approach. Ball tracking was first used in 2001 as a broadcast enhancement tool for cricket, but now we have developed state-of-the-art machine learning algorithms for both ball and player tracking. We are currently expanding the use cases to track not only people but also objects like bats and cars.
Edward：Hawk-Eye also places a huge amount of importance on the communication required for its adoption. We were closely involved with the initial stages of football VAR, leveraging our expertise in officiating technology to assist stakeholders in building protocols and procedures that have the maximum impact to the result of the game, whilst having minimal impact to the gameplay. The aim was to ensure fairer outcomes whilst maintaining high fan engagement and enjoyment.
Taro：Sony’s broadcast technology is also used at various sporting events, and as some of Hawk-Eye’s staff is always present at these sports venues, we have been able to gain feedback while providing support, enabling Hawk-Eye to develop better solutions for each sport. Another one of Hawk-Eye’s strengths is that we are always on site, listening to opinions 365 days a year, and we believe the expertise we have built up is one of the reasons that we are able to lead this industry.
Providing support for sports in areas other than officiating
──I understand that you are currently starting a new service?
Taro：Yes. Hawk-Eye’s real-time line-calling system (Hawk-Eye Live) is now being used at several international tennis tournaments to remove the need for line judges on-court. This is expected to increase players’ concentration levels through shortened and more efficient games, caused by the removal of challenges. Hawk-Eye will not replace umpires and referees and is a service only intended to support them, but we believe it could be used to make decisions in place of line umpires as a way of limiting the spread of COVID-19. Up to nine line umpires may be deployed at tennis matches, but by reducing the number of people on the court through judgements made by Hawk-Eye, we can help to ensure social distancing. In the 2020 US Open tennis championships, Hawk-Eye Live was used on 11 out of the 13 courts.
Edward：Following the outbreak of COVID-19, Hawk-Eye have started providing remote replay services for a new sports production flow, allowing a replay operator to use the system from their home to make live-to-air cuts. For golf matches, we have developed a system that links official data to video feeds and automatically tags which golfer is on which hole, allowing operators to quickly jump to the video feed they want to replay. This, and the fact that the operator can now provide broadcast replays from home on a 10Mbps consumer internet connection, significantly changes their workflow.
──Finally, could you tell us your future prospects?
Taro：Hawk-Eye’s services are currently used in over 20 sports, and, apart from ball sports, they are being used in athletics, horse-racing, and even motorsports. Ball tracking has started to be used for strategy analysis and scouting, and multi-angle viewing for matches shown on television and in videos is also under consideration, making it clear that the possibilities for Hawk-Eye are growing. “Sports definitely move people’s hearts, which lines up perfectly with our mission to do the same through technology. Making use of Sony’s technological capabilities, we will continue to provide new ways of watching and enjoying sports and boost the passion that sports inspires.
Edward：“Innovations” is a core part of who we are, and part of that is leveraging emerging technologies in new and different ways to provide value to our clients. We have the benefit of being in a position to get direct customer feedback, and there are so many amazing things going on throughout the Sony world that have direct or indirect applications in sport. Let us work together to bring about innovation in sports and we look forward to hearing any suggestions or proposals that you may have.
◆ ◆ ◆
About Hawk-Eye Innovations Ltd.
Hawk-Eye Innovations have been at the forefront of sports officiating and broadcast enhancement technology since 2001. Hawk-Eye’s vision-processing, video replay and creative graphic technologies to make sport fairer, safer, more engaging and better informed. Hawk-Eye Innovations form a key part of the Sony Sports Innovations Group with a shared mission to deliver best-in-class officiating, broadcast and digital solutions to the sports world.
Hawk-Eye applications
Tennis
Electric Line Calling (ELC)
Hawk-Eye’s tracking can accurately determine whether the ball has crossed the line or not, even for top tennis players’ serves, which can exceed 200 km/h. The system has incredibly high accuracy, with an error margin of less than 2 mm, and can also produce a CG replay in mere seconds by processing the tracking data. Challenges can have a great effect on the flow of a match, but by making the virtual replays visible to everyone, including the crowd and people watching at home through CG replays, matches will become much fairer. In addition, the graphics replays can be sold off as digital advertisement inventory for sponsors, creating further business opportunities for the event organizers.
Football
Goal Line Technology (GLT)
Football as a sport presented a number of challenges to overcome from an officiating and tracking perspective. Examples of these challenges include the need for officials to judge whether the ball has crossed the goal line in real-time and higher frame rate and resolution processing in real-time. Another key differentiator to previous ball tracking problems was that of ball motion, one of the hardest challenges within ball tracking was occlusion, where modelling ball flight isn’t always possible (as the goalkeeper may carry the ball over the line). This is coupled with the problem of noisy scenes where the ball may be obscured by the players and officials made ball tracking extremely difficult. To address all of these issues, the Hawk-Eye engineering team collaborated with Sony to innovate unique ball pattern recognition and orientation modelling algorithms that are used to determine the position of the ball in the image when only part of the ball is visible.
Video Assistant Referee (VAR)
Hawk-Eye‘s unique problem-solving expertise is also deeply ingrained into VAR technology, a technology at the forefront of Hawk-Eye’s offering, used in international football tournaments. The original issue with the introduction of VAR as a concept into football was communication. During the match, the referees and video assistant referees must constantly be in communication with each other using intercoms, deciding together when the best time to stop play is. Hawk-Eye addressed this inefficiency not only by setting up a system with multiple cameras to ensure appropriate angles for judgements, but also provide training for the referees who use the system, in addition to supporting referee training systems for qualifications and certification. Hawk-Eye have also held training camps with referees in preparation for VAR use at a selection of Japanese professional football league matches in the 2020 season.
Rugby
Television Match Official (TMO)
Hawk-Eye also provides video judgement assistance tools used by TMOs for rugby, where aggressive tackles and violent clashes are not uncommon. Depending on the matches, feeds from over 30 cameras may be aggregated, fed into servers, synchronized, and then displayed on the operation screen. Feeds that officials want to inspect in detail can be dragged and dropped, and judgements checked. Stills can be extracted, and manipulations such as zooms and slow-motion can be performed smoothly, allowing officials to check for illegal tackles, or whether or not a try was scored. Medical staff at rugby matches can use tablets to check where and how players have been injured, which contributes towards more accurate treatment. While increasing the safety of matches, better treatment also allows players to recover more quickly, providing great strategic and financial benefits for their teams.
Baseball
Video Judgment Support System
In Major League Baseball (MLB), there is a challenge system that allows managers to question the accuracy of the umpires’ judgements. By reviewing a video replay via Hawk-Eye’s SMART technology, managers determine whether or not to question the umpire. Hawk-Eye tracking systems have been introduced in 30 MLB ballparks.
Play Analysis Service
The MLB has also adopted Hawk-Eye’s sports tracking platform for play analysis in the 2020 season. Hawk-Eye’s vision-processing technology and optical tracking system are used to capture the movements of balls and players throughout the stadium with millimeter accuracy, and the data is analyzed. The synchronization and analysis of images captured by 12 high-resolution, high-frame rate cameras makes it possible to track three-dimensional skeletal models of players. This allows for real-time analysis of a player’s posture and movements enabling more precise evaluation of every play, including pitcher/batter mechanics, ball and bat tracking, and the movements of fielders and runners. The service is available in all 30 MLB ballparks and multiple training facilities under a multi-year contract. The data will also be reflected in MLB’s Statcast system, so it will be used during game broadcasts and allow fans to check the actual numbers on official MLB websites. In Japan, a verification test has begun this season with the cooperation of the Tokyo Yakult Swallows. Four cameras installed at the ballpark capture the pitches and batted balls between the pitcher’s plate and home base. The speed, rotation rate, direction of rotation and trajectory of the balls are converted into data. The number of cameras will be increased to eight at a later date to provide the same data as MLB.
