Sports Video Annotation: How to Build Training Data That Survives Match Day

Sports video annotation starts with the decision

How do you annotate sports video for computer vision? Start by writing the decision the eventual system must support—not by opening footage and drawing boxes. A clipping model needs event timing. A player tracker needs location plus identity through time. A pose model needs a fixed landmark definition. A field-registration system needs visible line geometry and a coordinate convention. Those are different measurement jobs, even when they use the same match video.

Labels are often called ground truth, but the phrase can hide judgement. An annotator still decides when an action begins, whether a partly hidden player remains present, whether a replay counts, and how to handle a camera cut. The first control is therefore an output contract: one document that links the sporting question, the observable label and the acceptable use of the result.

• Decision: what will a coach, analyst, editor or system do with the output?
• Unit: frame, timestamp, interval, object, track, keypoint, mask or field coordinate.
• Evidence: what must be visibly present before a label is allowed?
• Error cost: which false positive, false negative or identity swap matters most?
• Abstention: when should an annotator mark uncertain instead of forcing a class?

Gate 1: define one task before defining classes

The June 2021 SoccerNet-v2 paper is a useful warning against a universal annotation scheme. Its roughly 300,000 manual annotations across 500 untrimmed football broadcasts support separate tasks: action spotting, camera-shot segmentation, boundary detection and replay grounding. A timestamp useful for spotting a card cannot replace the spatial labels required to track the player receiving it.

Write one sentence in the form: given this footage, label this observable unit so this user can make this decision. Then list what the dataset explicitly will not support. If the first release is for detecting a ball-out-of-play event, do not quietly market it later as proof that the system understands possession, intent or tactical quality.

• Action spotting: mark an agreed reference moment and the event class.
• Temporal detection: mark a start, end and boundary rule for the action.
• Object tracking: mark location, persistent track ID and visibility state.
• Pose estimation: mark named landmarks plus visibility or occlusion.
• Field registration: mark pitch, court or track geometry in a declared coordinate system.

Gate 2: freeze the annotation specification

A sports video annotation specification should be short enough to use during a shift and precise enough to settle a disagreement. CVAT's living documentation allows a guide to be attached to a project or task so instructions remain visible to annotators. The software feature is not the principle: the important control is that every labeler works from the same versioned rules.

For each class, include a positive example, a near-miss, an exclusion and the exact boundary rule. State how to treat warm-ups, substitutions, replays, slow motion, crowd shots, picture-in-picture, score graphics and post-event reactions. For tracked objects, distinguish occluded from outside the frame, and decide whether an identity survives a temporary disappearance or camera change.

Do not rewrite the guide silently after production begins. Record the change, the affected labels and whether earlier work must be reviewed. A dataset with a stable label name but changing meaning is not one dataset; it is several incompatible versions sharing a folder.

Gate 3: sample the conditions the system will face

Choose footage by operating condition before choosing frames. A football tracker must encounter crowding, similar kits, wide and tight shots, substitutions and broadcast cuts. Basketball adds dense overlap and fast direction changes. Tennis introduces a small fast ball, court-line boundaries and side changes. Cricket combines long shots, close-ups, replays and equipment occlusion. Formula 1 footage switches between cars, onboard cameras, pit lane, timing graphics and trackside views.

Research datasets show why coverage matters, but their scale is not a purchasing target. SportsMOT, first posted on 11 April 2023 and published at ICCV 2023, contains 240 clips, more than 150,000 frames and more than 1.6 million player boxes across football, basketball and volleyball. The authors identify variable-speed motion and similar-looking athletes as central association challenges. SoccerNet-Tracking, published in the June 2022 CVPR Workshops, uses 200 thirty-second sequences plus a complete 45-minute half for long-term tracking.

Those projects demonstrate task design under research conditions; they do not prove how many labels a club or publisher needs. A smaller dataset can be useful if it covers the real camera, venue, sport and error slices. A large collection of nearly identical daylight frames from one match cannot establish match-day robustness.

• Create a coverage matrix for venue, camera, lighting, zoom, game phase and broadcast treatment.
• Reserve difficult cases deliberately instead of waiting for random sampling to find them.
• Track class frequency by match and condition, not only across the full frame count.
• Keep source rights, permitted use and provenance attached to every video group.

Gate 4: match label geometry to the output

Use the cheapest geometry that answers the declared question. A timestamp may be enough for a card or pit stop. A bounding box can support detection and tracking. Keypoints suit pose questions. Masks add contour detail but also add boundary judgement and review cost. Field lines and calibration points are required when pixel locations must map to a real playing surface.

Video tools can interpolate object tracks between manually set keyframes. CVAT's current documentation describes interpolation as a workflow in which the annotator labels keyframes and intermediate locations are generated linearly. That saves effort only when the motion between keyframes behaves as assumed. Camera cuts, sudden accelerations, occlusion and direction changes still need human inspection and corrective keyframes.

Do not use label complexity as a quality signal. A precise mask can be consistently wrong, while a simple event tag can be fit for purpose. Quality is the agreement between the rule, the visible evidence and the downstream decision.

Gate 5: measure disagreement before scaling

Run a representative pilot through at least two independent annotation passes or a separate reviewer before expanding the job. The goal is not to manufacture a high average score. It is to find where competent people interpret the same rule differently. If disagreements cluster around replay boundaries, partial visibility or the distinction between two actions, revise the rule before buying more labels.

CVAT's living QA documentation describes consensus replicas, validation subsets, ground-truth jobs, review and quality reports. Availability varies by product and plan, so these are examples of controls rather than a recommendation to buy a specific CVAT tier. An equivalent workflow can be run with exported files and a separate comparison script.

Measure the error that belongs to each label type. For event tags, compare class and timing tolerance. For boxes, compare localization and missed objects. For tracks, count identity switches and broken tracks. For keypoints, record visibility mistakes separately from position error. A single blended quality score can hide the failure that matters most.

• Disagreement log: clip, label, annotator decisions, reviewer decision and rule change.
• Gold set: representative examples locked before production scoring begins.
• Slice report: quality by camera, venue, class, occlusion and game condition.
• Stop rule: pause scaling when ambiguity comes from the specification, not the annotator.

Gate 6: protect identity and time

Sports tracking fails differently from ordinary object detection. The label must say whether the person or car seen now is the same entity seen earlier. Define when a track starts, when it ends, what happens through occlusion, whether re-entry receives the old ID, and whether identities may bridge a broadcast cut. If jersey number, team, role or car number is uncertain, store uncertainty rather than inventing a value.

The June 2024 SoccerNet Game State Reconstruction paper illustrates how demanding that chain can become: its 200 thirty-second sequences include pitch-line points for calibration and athlete positions with role, team and jersey-number information. Those fields are task-specific research labels, not a universal schema or a minimum requirement.

A preprint posted on 16 August 2026 reports that long-term sports tracking still has to manage frequent occlusion, rapid camera motion and athletes disappearing and reappearing. Its performance claims are author-reported preprint results and were not treated here as independently replicated evidence. The durable lesson is narrower: identity policy must be explicit in the labels because the model cannot repair an undefined identity rule later.

Gate 7: split by match before training

Adjacent video frames are highly related. Randomly mixing them across training and test sets can let the same kit, camera, venue, scoreboard and passage of play appear on both sides of the evaluation. The resulting score may measure memory of one production environment more than transfer to a new match.

Split the data at the highest practical source group: match, session, race, court feed, venue or camera installation. Scikit-learn's current GroupKFold documentation formalizes the general rule by keeping the same group out of both training and testing folds. Using a match ID as the group is our sports-video application of that method, not a claim that one split design fits every project.

Keep a separate release ledger listing every source group, its assigned split and the reason. If the deployment target is a new venue or broadcast provider, reserve an evaluation slice that actually tests that change. Never tune the annotation rules repeatedly against a supposedly hidden test set.

The seven-gate release record

Before an annotation release reaches model training, the owner should be able to produce seven artifacts. Together they turn sports video annotation from an invisible subcontracting step into an auditable measurement workflow.

NIST's AI Risk Management Framework, released on 26 January 2023, is cross-sector guidance rather than a sports annotation standard. Its relevance here is the requirement to connect measurement, validity, reliability and risk to the intended context. It does not certify a dataset, set an acceptable error rate or replace sport-specific judgement.

• 1. Output contract: user, decision, observable label, exclusions and error costs.
• 2. Specification: versioned classes, examples, boundaries, visibility and uncertainty rules.
• 3. Coverage matrix: matches, cameras, venues, conditions, classes and difficult slices.
• 4. Review report: disagreement, corrections and quality by label type and condition.
• 5. Identity policy: track start, occlusion, re-entry, camera cuts and unknown values.
• 6. Split ledger: source groups assigned to train, validation and untouched test data.
• 7. Dataset card: provenance, rights, permitted uses, format, versions and known gaps.

Limitations and source notes

This is a desk-researched workflow, not a benchmark of annotation vendors or software. The Sports AI Review did not label a proprietary match dataset, time annotators, compare platform prices or train a model for this article. No universal label volume, agreement threshold or acceptable error rate is claimed; those decisions depend on the task, consequences and deployment conditions.

The SoccerNet and SportsMOT publications are research examples with specific sports, footage and annotation schemes. Their dataset sizes are historical facts, not recommended minimums. CVAT and scikit-learn pages are living product documentation checked on 27 October 2026; where the pages did not display a publication date, this review does not present the check date as one. Product features and plan availability can change.

Evidence was checked through 27 October 2026. The newest research item used is the McByte++ preprint posted on 16 August 2026; it is cited only for its documented problem setting, and its benchmark claims remain author-reported. This guide contains no medical claim, athlete-selection recommendation, competitive prediction or betting advice.