AI Judging in Sports

There are things in life that are subjective, like beauty, taste, emotions, and feelings. However, when it comes to judging in competitive sports, decisions have become a lot more cut and dried, thanks to the use of artificial intelligence (AI) systems.

Several sports organizations have been using AI to judge certain aspects of their competitions and games for years, but as the systems become more sophisticated, more are jumping on the bandwagon.

The board of the Premier League (the highest level of the English football (soccer) league system) last April voted unanimously to introduce the use of “semi-automated offside technology.” The new system will be used by the League for the first time in the 2025 season.

“The technology will provide quicker and consistent placement of the virtual offside line, based on optical player tracking, and will produce high-quality broadcast graphics to ensure an enhanced in-stadium and broadcast experience for supporters,’’ the League said in a statement.

AI and the human factor of judging.

The Hawk-Eye computer vision system made its tennis debut in 2003 for broadcasting purposes, but was approved in 2005 after a notorious U.S. Open Tennis match between Serena Williams and Jennifer Capriati in 2004, during which Williams was the victim of multiple bad calls in the third set and went on to lose the match.

Use of Hawk-Eye was expanded during the COVID-19 pandemic, and the 2020 U.S. Open was played without line judges on all but two of the main courts. Since Hawk-Eye has been in use, between 190 and 200 judges have been replaced, depending on the stage of the tournament, says Sean Carey, managing director of competition operations, at the U.S. Tennis Association (USTA).

“The reason we bring technology in for this level of tournament—and we want to do it across every level if we could afford it—is to ensure integrity and the fairest and most even calls,’’ Carey said.

Hawk-Eye, which uses cameras to track the trajectory of a ball and create a three-dimensional (3D) representation of it, is now being used by 23 of the top 25 global sports leagues and federations, according to the company. Yet, the sentiment appears to be that AI will never fully replace human judges.

This has been the subject of much debate in Major League Baseball (MLB), a sport grounded in tradition, noted Daniel Martin, an associate professor of economics at the University of California at Santa Barbara. MLB is using Hawk-Eye to automatically monitor strike zones, and is questioning whether to get rid of umpires, given that the system is “incredibly accurate,’’ he said.

Yet, Martin said he doesn’t think that will happen, because society is not ready to fully give way to machines to judge sports. “There’s something deeply appealing about the human element [in a game] and this is why we end up with AI oversight,’’ he said. “People like people in the mix.”

While baseball matters in an economic sense, we also have to factor in people’s emotional experience with the game, he said. Hawk-Eye is useful when calls are challenged to make the ultimate decision without human bias. At the same time, humans need to remain in the loop to make the most of the judgment calls, Martin said.

David Almog, an economics Ph.D. candidate at the Kellogg School at Northwestern University and the lead author of the 2024 paper, “AI Oversight and Human Mistakes: Evidence from Centre Court,” said human behavior changes when technology is in play. Almog worked with Martin and others to analyze a set of Hawk-Eye data on how umpires called matches.

The researchers found that umpires’ accuracy improved, and their overall mistake rate declined by 8%, after Hawk-Eye was introduced. Yet, oddly, there were instances where tennis umpires’ mistakes increased after AI was used. This suggests to Almog and Martin that the umpires were feeling the pressure of the AI system and reacting to it, whether consciously or not.

The psychological pressures on human judges when AI systems are used is cause for concern, Almog said. The fact that Hawk-Eye can overrule a line umpire is good for the game and gives umpires the impetus to improve, which is also good.

“It’s a good thing if all you care about is improving performance,’’ he noted, “but if you care about [a human’s] welfare … that’s still an open question.”

The solution, said Martin, would be to have a couple of human arbitrators, although AI serves as a neutral arbiter.

Ultimately, Martin said, sports will do what makes the most business sense. That said, he believes humans will remain in the mix “because people want to see humans,” and there is the entertainment value of seeing an umpire make a mistake. Even watching AI overrule an umpire is entertainment, Martin said.

“What we’re selling with sports is not perfection. What we’re selling is the human experience—but we want some kind of fairness.”

Increasing predictions, levels of probability.

While the PGA Tour has been using machine learning tools for many years, the system was revamped about two years ago, said Ken Lovell, senior vice president of golf technologies. Unlike the square field used in many other sports, it has taken the PGA years to map its golf courses in three dimensions, he said.

The real-time, proprietary ShotLink system is comprised of logistical information, statistical data that is translated into something humans can understand, as well as a networking layer, and sensors, which are new this year and which Lovell refers to as “the cool piece.” Three different types of sensors are embedded in the ground and cover the golf course, including “military-grade radar” that provides data about every shot.

There are also 12 cameras and a group of people who watch for shots that go outside of the course parameters. The sensors and cameras provide data and are used to predict the location of a shot before it hits the ground, he said.

“It’s not just about predicting the bounce and roll out; we can tell you where [the ball] will end up with a level of probability around everything that happened in real time for every shot on the golf course,” Lovell said. “In addition, we can look at obstructions and tell you the probability of where the next shot will likely be able to be hit.”

Golf is unique in that players call their own fouls, he added. The rules officials on the ground provide guidance, and the idea behind using AI is to “give tools to the rules officials to help them do their jobs,’’ he said.

Lovell said there is a lot of “killer math’’ going on in the background to trace a ball in real time using data from the sensors, radar, videos, and cameras that is sent into the cloud.

“Sometimes, it’s hard for players to see a shot,’’ and there can be a 30-yard or 40-yard discrepancy, Lovell explained. In that instance, a rules official gets involved, since “they know the rules of golf better than anybody,’’ he said.

Right now, the PGA is building a system that will allow rules officials to look at all of this information on a tablet. It will have a view from every camera that saw the shot, enabling an official to zoom in or pan out as much as they want, he said.

Because the golf course has been painstakingly mapped, there is information on the outlines of a water hazard line. “I can draw in space a vertical plane from the edges of that line and can tell … exactly where the ball crossed the line,” Lovell said.

The International Gymnastics Federation (known as FIG) also is bullish on the use of AI, although the idea started as a joke, when Masanori Fujiwara, then a project leader at Fujitsu, met with Morinari Watanabe, who was head of the Japanese Gymnastics Association, and jokingly said that “In the future, maybe robots will be scoring.” Fujiwara took the joke seriously and built a prototype, which led to the development of the Judging Support System (JSS) in partnership with FIG in 2017.

JSS was used to judge the pommel horse, vault, and rings events at gymnastics’ 2019 World Championships.

Proponents of JSS say it can eliminate biases and make the sport fairer. But, as is the case with the other sports, there is also debate about whether it will take away something; in this case, the subjectiveness that factors in artistry and performance as part of a competitor’s score.

JSS has been enhanced, and now uses camera-based imaging instead of sensors. The reason, says Fujiwara, now general manager of the human digital twin business division at Fujitsu, is that “at the time, the Microsoft Kinect was in the spotlight as a skeleton recognition technology, but it had the inherent problem of not being able to achieve the 6m+ range performance required for gymnastics and other sports.”

Meanwhile, Fujitsu Laboratories was developing Lidar (light detection and ranging) for autonomous cars and determining terrain for heavy equipment work, he said, explaining that developers thought “the Lidar could measure distances of 10m or more and that by speeding up and improving the resolution of laser scanning technology and developing the angle of view control technology, it could be acquired as a depth image capable of detecting complex human movements in many sports, such as gymnastics.”

Later, as development progressed, Fujitsu “began to feel Lidar’s limitations and moved on to cameras,” Fujiwara said.

The technology has been refined and in 2023, JSS officially operated for all 10 apparatus used at the Antwerp World Championships. “Through the development of JSS and the actual use of JSS by FIG, we have been more confident that new value can be created by digitizing human movement,’’ Fujiwara said.

In the near future, Fujitsu will roll out its Human Motion Analytics platform, which uses technologies developed for JSS, including its motion constraint corrector, a correction algorithm that can significantly reduce estimation errors in posture recognition, according to Fujiwara. “Until now, posture recognition blurring has been an issue in deep learning image recognition, so the motion constraint corrector technology enables more accurate skeleton recognition,’’ he said.

The corrector reduces jitter by ensuring skeleton length is as constant as possible and preventing joint position and angle abnormalities. “In gymnastics (JSS), the relative position of the head, legs, and so on, determines whether the technique is completed,’’ Fujiwara said. “In the case of the Human Motion Analytics Platform, the aesthetic elements of human movement are also important because of their relative position, such as the position of the head and the position of the legs, which can help improve the judgment of human movement.’’

Fujiwara said he expects JSS will be used in other sports because of its ability to instantaneously capture complex, high-speed movements.

Olympians still on the fence.

Not every sport has climbed on the AI bandwagon yet. The International Olympic Committee (IOC), which established a working group in 2023, made no mention of using AI systems to judge the various sports at the 2024 Summer Games. In an April statement, the organization said it is “still at the beginning of its AI journey,” with a plan to “leverage learnings from the Olympic Games Paris 2024 and other Olympic events to identify AI solutions that will improve the operational efficiency and sustainability of future Olympic Games.”

As to whether colleges and universities will use AI to judge competitive sports, Natalie Kupperman, an assistant professor of data science at the University of Virginia, said there tends to be a trickle-down effect. “I would not be surprised if we see specific camera technologies implemented in the college atmosphere,’’ she said, adding that the issue is the cost of outfitting the arenas and stadiums where college sports are played.

Northwestern’s Almog said he hopes that as the use of AI systems for judging sports increases, the human element will be considered. AI systems also introduce the potential for distortions, sometimes referred to as AI hallucinations, when models produce misleading results.

“If you introduce distortions you run the chance of AI oversight not improving the way you thought it would,’’ Almog pointed out. Humans will rationalize the change in their behavior based on the AI mechanism in place, which may be good for them, but not necessarily good for the sports organizations that brought in the systems. “So they’re acting in different interests,’’ he said. “That’s the cautionary tale.”