With defending champion Rafael Nadal serving for the match against Karen Khachanov in Round 3, the Russian earned an opportunity to level the fourth set at 5-5. After spending the first six balls of the break-point rally defending from sideline to sideline, Khachanov whipped a dipping backhand at the feet of an approaching Nadal. The world No. 1’s shoestring volley landed just inside the baseline, but the floating reply gave his opponent plenty of time to set up a down-the-line forehand pass.
Leaving Nadal stranded, Khachanov breathed new life into a match that was already four hours old. As the Arthur Ashe Stadium crowd rose to its collective feet, he celebrated with a double fist pump.
Here’s how IBM Watson saw the action, on its 0-1 scale: The point received a 1.00 for crowd cheering, a 0.85 for match analysis (a relative measure of the point’s importance) and a 1.00 for player gestures, all contributing to a 0.97 score in overall excitement.
It may seem impossible to pick out the most exciting point from the 2018 US Open – let alone that four-hour, 23-minute epic – but IBM Watson zeroed in on this one. The artificial intelligence tool is in its second year at the US Open. If it was a grade-schooler last year, it’s a well-polished young adult now, impressive but with room yet to grow.
In 2017, the technology was used to produce “Cognitive Highlight” packages, which simply picked out the top points from the winning player in a given match. This year, the renamed “AI Highlights” are also able to tell the full story of a match in a roughly three-minute package, featuring both its most exciting and most important points.
"We started this about a year-and-a-half ago and maybe even before that, with analyzing audio and photos, and then we moved into video," explained Stephen Hammer, Sports Chief Technology Officer and Distinguished Engineer at IBM. "We started out by picking out exciting moments, and once we could do that, then it became, 'What can we do with that?' Putting those exciting moments into really compelling highlight videos that would otherwise take a person several hours to do, we can do in a very short period of time."
Hammer described how year-two enhancements have allowed for more accurate and thorough analysis, which ultimately produces a more thoughtful finished package.
Facial recognition has been improved to further inform other body language cues that may – or may not – tip off a turning point in the match. For example, if a player is frowning when he or she makes a fist-pump motion, that player may, in fact, be asking for a towel. Another improvement relates to crowd noise. Fan reaction is a great barometer of excitement, but favoritism can skew the results. To counter this false positive, IBM Watson now uses individual player thresholds to equalize the fan-favorite effect, rather than a single baseline for both.
The presentation of the highlight packages is now smoother, too, with more precise clip start and end times, and dynamic bumpers to provide context pre- and post-roll and in between sets.
The IBM technology is part of a larger industry trend of managing media assets and using AI to help scale and increase delivery speed. Through the end of Day 11, over 30,000 points and 360 hours of play have been analyzed, with 174 highlight packages “automagically” produced.
While this functionality certainly makes the jobs of video and other digital teams easier, it is also paying dividends in the coaching ranks. During the 2017 US Open, IBM staffers met with the USTA to discuss how Watson could assist with player development. Now, AI has taken over the time-intensive “match tagging” process that logs each point in a match, shortening what used to be a 24- to 48-hour turnaround time per match down to just 30 minutes or less.
“Working with IBM enables us to process and index video, using AI to free up valuable intellectual capital that we can re-allocate to more interpretive and customized analysis," said Martin Blackman, General Manager, USTA Player Development. “We are excited that, working together with IBM, we can create a new solution that will revolutionize our ability to pair analytics with coaching expertise to drive performance."
The technology breaks down each point with information that includes how the point ended (ace, forehand winner, backhand error, etc.), serve depth and location, and score, as well as movement-based metrics like distance run. Looking ahead to the future, there is a lot more to come. Eventually, the hope is that coaches will be able to run a query to view, for example, every cross-court backhand hit in a match, or every forehand that landed within five feet of the baseline.
“Being able to get to an individual shot level is one of the places we want to take it,” Hammer said. “There really is a lot we can do with it.”
Hammer also foresees AI analysis growing to include instances when the ball is not in play, looking at the body language and emotion of players in between points and during changeovers.
As the US Open continues to grow – in its 50th anniversary edition, the tournament set three seperate single-day attendance records – the technology will continue to improve along with it. Though it was not included among the record 71,902 fans that flooded the USTA Billie Jean King National Tennis Center grounds on September 1, IBM Watson is always watching.
