Developing MLB’s Automated Ball/Strike System (ABS)
Written by Perry Pierce, Matthew Whitrock, and Stuart Cheshire
Early in 2019, Major League Baseball announced a partnership with the Atlantic League of Professional Baseball (ALPB) to test new playing rules in order to observe the effects of potential future rule changes and equipment. One of those initiatives was the creation and testing of an automated ball and strike calling system (ABS). The goal of the tests was to validate whether ABS was able to make and consistently communicate the correct call to the umpire quickly enough for the umpire to make the call on the field without introducing a delay.
The results of our work in 2019 confirmed the system was able to work with sufficient consistency, reliability, and responsiveness, which enabled us to look at expanding the system for the 2020 season. Unfortunately these efforts were put on hold due to the COVID-19 pandemic. With the resumption of the Minor Leagues and the ALPB in 2021, we are ready to start testing again. In this article, we will explain how ABS operates, review the system’s initial design and subsequent iterations, and preview some of the additional ABS changes and improvements slated for 2021.
Television Graphics and Real-Time Strike Zones
In today’s game, many broadcasters use either a “live” strike zone graphic projected directly above home plate or a standalone strike zone graphic elsewhere on the broadcast. A similar graphical representation of the strike zone is used in MLB’s Gameday product and the MLB App. In each case, the strike zone graphic depicts the pitch relative to the boundaries of the strike zone.
An example of these renderings can be seen below from one pitch in a 2019 game between the Detroit Tigers and the Boston Red Sox.
While there is a slight delay between the catcher receiving a pitch and the pitch location appearing on-screen, the broadcast graphic consistently updates faster than the umpire makes their call. An example of this is below and illustrates the window that ABS operates in: there is a period of time where the relevant information about the pitch is known but the umpire has not yet made a call, which gives ABS an opportunity to make a determination and relay the call to the umpire.
The history of showing pitch locations on TV in near real-time granted us a conceptual baseline from which to work. Showing a live strike zone on air requires some basic ability to identify a broadcast-appropriate strike zone, determine the location of the ball within that space, and render each in a reasonably convincing manner.
Theory of Operation
The primary inputs for ABS are the pitch arc — the path the ball takes as it proceeds from the pitcher to the catcher — and the location and dimensions of the strike zone. The pitch data generated by the MLB tracking system is composed of a set of polynomials describing the path the center of the ball travels through space in MLB’s standard coordinate system, where the y-axis points toward the pitcher’s mound from the back of home plate, the z-axis points directly up from the back of home plate, and the x-axis is orthogonal to the other two axes.
As soon as the pitch data has been generated by the tracking system, ABS uses that information and the current strike zone definition to determine whether the pitch is a strike. The strike zone definition has varied across different iterations of the ABS system, but as a general rule, ABS constructs a two-dimensional shape on the x/z plane at a specific depth and checks whether the ball intersects that shape. The ball is modeled as a circle centered at a point along the pitch arc with a radius of 1.45 inches as defined in Rule 3.01 of the Major League Rulebook. The result of the ball/strike evaluation is then relayed to the umpire; we’ll speak more on how that is done later in this post.
Constraints and Limitations
The single most important requirement for ABS implementation is the accuracy of the pitch arc being generated. To this end, MLB performs ground-truth testing at every ballpark to evaluate the accuracy and precision of in-stadia tracking systems; the methodology for this was well-covered in an MLB Technology Blog post by Clay Nunnally in December 2019.
Just as important as accuracy for ABS is latency. Currently, there is some intentional delay between when the pitch is received and when the umpire makes their call, which allows the umpire to take a brief moment to make their decision. That brief moment is the window in which ABS must consistently deliver a call to the umpire; anything longer than that and the umpire has to make the call themselves to avoid delaying the game.
The other main constraint is tracking system reliability. In order to effectively standardize the called strike zone, it is not sufficient to quickly and accurately relay a ball/strike call to the umpire some of the time, or most of the time; that goal must be achieved all of the time. It would not matter that 50 percent of pitches are called instantaneously with perfect accuracy if some percentage of pitches are either not tracked at all, significantly delayed in producing a tracking result, or poorly tracked. MLB spends significant effort monitoring the tracking system to ensure it is producing timely and accurate results.
Finally, there are some cases where, even in a game where ABS is operational, the umpire will make the final determination of a ball or a strike being called based on criteria other than the strike zone. ABS is specifically focused on the strike zone and the flight of the ball, so the umpire retains personal responsibility for judging the other criteria that help determine pitch outcomes, such as checked swings or catcher’s interference. Using ABS allows the umpire to be better focused on anything that would result in a decision without consideration for the actual pitch location.
Building the Initial System
After the Atlantic League’s partnership with MLB was announced in early 2019, MLB tracking systems were installed and ground truth tests were conducted at each Atlantic League venue during April and May of that year. Once systems were operational, ABS was tested by running the system in the background without attempting to use the system on-field at several venues to identify and correct any critical issues prior to full rollout of ABS in the second half of the 2019 Atlantic League season.
The ABS system design has four discrete subsystems:
- The MLB tracking system;
- An interface the ballpark operator uses to set the correct batter;
- An MLB server that receives the tracking data and has the ball-strike evaluation code;
- A low-latency communication system to relay calls to the umpire.
All together, the system design looks like this:
As each batter came to the plate, the operator updated the system to ensure the correct batter was selected; this information was relayed to MLB tracking servers so that the Field Data Collector (FDC) knew the correct strike zone to use when the next pitch was thrown. When the pitch was thrown, the MLB tracking system published pitch data to a publish-subscribe queue; FDC read that data from the queue, determined whether the pitch was a ball or strike, and published that result to a pub/sub topic along with any other data it was responsible for collecting and publishing. The operator console was responsible for receiving the message published to the topic, indicating the call to the ABS operator, and playing the appropriate audio file for that call. The audio-out port of the ABS operator’s laptop was wired to a walkie-talkie; when audio played, the walkie-talkie transmitted audio for the duration of the audio clip and then ceased transmission.
Initial Results
ABS was used in the Atlantic League for the second half of the 2019 season, but we spent the first half of the season proving feasibility, collecting feedback from players and umpires, and creating runbooks for in-stadium personnel. One early area of feedback was the height of the strike zone, and a second was the reliability of the call in the umpire’s ear.
The ABS strike zone shape we tested in the Atlantic League was a strike zone top and bottom sourced from MLB’s Stats API, which has player-specific strike zones for players who have appeared in MLB games but otherwise uses default strike zones based on a player’s height. To evaluate the strike zone across the full depth of the plate, the strike zone was modeled as a series of two-dimensional shapes on the x/z plane at specific intervals, conforming to the outline of the plate. A pitch could be above the top of the zone at the front of the plate but descend far enough to intersect one of the shapes closer to the back of the plate. Players noted that this produced a taller strike zone than they expected.
During initial testing, the design of the walkie-talkie system caused occasional delays in getting the call to the umpire. A number of alternatives were tested, but over time it became clear that a modified off-the-shelf walkie-talkie transmission consistently added an unacceptable amount of latency to the system and a different approach for relaying the call to the umpire would be necessary.
For the first live-game use at the 2019 Atlantic League All Star Game in York, PA, we replaced the walkie-talkie system with an iPhone which the umpire wore on their belt. To play the sound in the umpire’s ear, we used an AirPod wireless earbud so the umpire did not need to be wired up to a device.
We leveraged the iPhone’s browser and created a simple browser application that listened to the messages broadcast by FDC over a publish-subscribe topic; upon receipt, the appropriate call would play in the umpire’s earpiece. This removed the walkie-talkie transmission latency, and while overall system performance did improve there were still intermittent delays of one to two seconds for umpires to hear the ABS call.
We found the AirPod worked sufficiently well in our testing, but umpires voiced concerns over earbuds falling out when removing their mask or becoming gradually unseated from their ear during play due to sweat, movement, etc. Due to that feedback, we migrated back to a wired security earpiece that connected to the iPhone, with multiple sizes of silicone earbud sleeves to better fit each umpire’s ear. Migrating back from a Bluetooth headphone to a wired one also removed an additional source of latency.
Arizona Fall League (AFL) Design Iteration
For the 2019 AFL season, we made a few important updates. First, the MLB tracking system at Salt River Field was identical to the ones used in all thirty Major League ballparks in 2018 and 2019. In addition, we updated how we used the iOS devices in order to address the intermittent latencies that were noted in the Atlantic League sites. We stuck with the wired earpiece as that was overwhelmingly favored by umpires.
To address the intermittent latency when ABS ran in the iOS browser, we developed a dedicated application that performed the same functions as the web app. Switching to a dedicated app eliminated the client-side delays we had seen with earlier approaches and consistently delivered the ball or strike determination to the umpire’s ear on-time.
With these results in hand, we were confident that we had sufficiently addressed operational issues and were ready to scale up the system for broader use.
2021 and Beyond — Scaling Up ABS
Taking into account the progress made in 2019, MLB determined that ABS is ready to be taken to additional leagues and parks in preparation for possible use at the highest levels of Minor and Major League Baseball. Like everyone else, ABS needs to work its way up to “The Show”! Therefore, in addition to the Atlantic League, we have installed ABS at several venues in the Low-A Southeast League.
The major benefit of testing ABS in the Low-A Southeast League is that several of these venues have also been equipped with the latest revisions of MLB tracking technology, so ABS’ overall system design in these venues will more closely mimic what the system would be in Major League ballparks.
Conclusion
All in all, 2019 was an amazing year for the team that designed, developed, and operated ABS. Engineers from across the MLB Technology organization and members of MLB’s Baseball Operations teams banded together to make this project a reality, going from concept to live in-game use in less than five months. In recognition of the effort, the MLB team was recognized with the 2019 SportTechie Award for Innovation of the Year for ABS’ potential to “transform the governance of baseball’s most fundamental interaction”.
Despite losing the opportunity to test in 2020 due to COVID-19, the team is ready to continue testing the system in the Low-A Southeast League and the Atlantic League this spring. Our goals for the year are to validate system performance and reliability across a larger number of games and make refinements to our design where necessary to harden the design for all ballpark conditions. We look forward to further exploring how ABS and other technologies can enhance the game on the field for the greater enjoyment of all who follow the sport.
from Hacker News https://ift.tt/3w9sH12
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.