How ReCAP Reads Statistical Overlays in Election Coverage

Election night compresses changing numbers, party labels, candidate names, district maps and sponsor messages into a dense visual stream. A lower-third saying “Seat: 84” may coexist with a national swing chart, a scrolling results ticker and a map of marginal seats. For archivists, analysts and live desks, these are not interchangeable objects: one is a statistical overlay, while the others provide context, identity or branding. ReCAP treats that distinction as the first step toward turning moving broadcast images into structured evidence.

ReCAP is an EU-funded research initiative for Real-time Content Analysis and Processing. Its tools are designed for broadcast-quality video, including metadata extraction, quality monitoring, face and logo recognition, and duplicate detection. Applied to election coverage, the same foundations let a system find an overlay, determine whether it displays a measured result, extract its text and preserve the moment when it appeared. The result is a searchable timeline rather than a collection of screenshots that quickly become difficult to interpret.

From Video Frames to Text Candidates

A ReCAP workflow begins with the video feed, not the producer’s graphics template. Frames retain their timecode, resolution and frame rate while the system searches for text-shaped regions that survive compression and scaling. This matters in Australian archives, where the same count may arrive through a broadcast master, a web stream and a clipped social video at different dimensions. A time anchor lets the system reconcile those versions and identify the master feed carrying the cleanest overlay.

Candidate regions can be found through contrast, edges, connected strokes and temporal motion. A lower-third grows or changes between frames, while a permanent studio caption remains stationary; both behaviours help separate temporary graphics from the set. Optical character recognition then reads the localized region, and a tracking step gives the same overlay a stable identity as its values update. Bounding boxes are recorded in normalized coordinates so the location remains meaningful after a portrait clip is converted to a landscape player.

The project’s related work on counting on-screen text overlays addresses a basic but consequential question: how many distinct overlays are visible, rather than how many text boxes an OCR engine happened to return. Two boxes for one two-party-preferred panel should count once, while a split screen can contain several independent tallies. Counting therefore depends on grouping text by panel, hierarchy and persistence, with uncertainty retained when graphics overlap or change too quickly for a confident match.

Classifying Statistical Graphics

A statistical overlay carries a measured claim about an election: votes, seats, turnout, swing, preference share, margin, spoiler vote or forecast range. ReCAP can label the visual form first—scoreboard, chart, map, ticker, badge or table—and then recognise numeric tokens and their units. Labels such as “per cent”, “votes” and “seats” are retained because the same digits can represent incompatible quantities. Repeated layouts also provide templates that help the classifier recognise when a new number is an update to a known graphic.

That classification must remain separate from the surrounding broadcast. Search and crawl systems can encounter unrelated commercial pages, and a page presenting safe casino bonus information may use large percentage badges or rounded panels without having any electoral meaning. Such material belongs to a different evidence class, even when its design resembles a results card. Source context, programme boundaries and extracted semantics jointly prevent a flashy graphic from being indexed as a seat count.

Several election formats complicate the task. Panelists can appear beside a map, a lower-third can obscure part of a total, and a network may replace one national figure with several state figures in the same corner. ReCAP combines layout, motion and language cues, then attaches a confidence score to each classification rather than forcing every case into a perfect label. A low-confidence result can be queued for human verification, which is especially valuable for tiny regional margins and rapidly flashing changes.

Turning Overlays Into Searchable Metadata

Each accepted overlay becomes a record with its source identifier, start and end time, bounding box, recognised text, graphic type, confidence and related entities. Election-specific tags can distinguish a displayed seat count from a forecast, a declared result from a notional tally, or a primary vote from a two-party-preferred figure. Keeping the raw transcription beside the normalised value allows an editor to inspect what the screen actually said without mistaking the parser’s interpretation for the broadcaster’s wording.

ReCAP’s face and logo recognition adds another layer. A face match may associate an on-screen candidate with a candidate record, while a party or broadcaster logo can identify the source of a graphic. Duplicate-content detection can reveal that the same tally has been reused across programmes or streams. These links are useful for retrieval, but they remain annotations rather than proof: visual resemblance does not establish identity, and repeated footage does not prove that two updates are the same event.

The metadata layer also needs provenance rules for values that are calculated rather than displayed. If an overlay shows “54.2%” and an election model estimates seats from it, the model’s output must not be merged with the observed claim. Each value should retain its denominator, geographic scope, election stage and timestamp. During a close count, two screens may legitimately show different totals because they use different cut-off times; storing those details allows comparison without rewriting history.

Australian Election Night in Practice

Australian viewing habits make this visually rich environment especially common. In Sydney and Melbourne, many people follow a count on a phone during the evening commute while the main television service carries commentary or a debate replay. Regional viewers may rely more heavily on broadcast and catch-up services, so the same result can be consumed through several interfaces. ReCAP’s normalised coordinates and stable timecodes support a unified search across those viewing contexts, including clips shared the next morning.

The local media market combines public broadcasters such as the ABC and SBS, commercial free-to-air services including Seven, Nine and Network 10, and news channels that compete for attention around major events. Network branding, election maps and count widgets often use similar palettes, while streaming products add captions and player controls around the video. A robust overlay detector must treat those interface elements separately from editorial statistics and remain readable when a wide television frame is compressed into a mobile feed.

Australian electoral information also has a specific legal context. Results and formal status should be attributed to the Australian Electoral Commission or the relevant state electoral commission rather than inferred from a graphic alone. Broadcasters must meet disclosure and electoral-communication requirements under Commonwealth and state law, including the Electoral Act 1911 where applicable. Where the Privacy Act 1988 and Australian Privacy Principles apply, face templates, inferred preferences and other personal information need careful governance; recognising a face in a frame does not make every resulting inference publishable.

Commercial breaks add another layer of visual competition. Australia’s sizeable regulated gambling market produces frequent sports-betting and online-casino advertising, governed by federal rules such as the Interactive Gambling Act 2001 alongside state codes and advertising standards. Election graphics may share bold badges, gradients and percentage symbols with those advertisements. Programme segmentation, sponsor labels and a separate advertising taxonomy help ReCAP distinguish a political result from a promotional offer even when both appear within the same evening’s transmission log.

A Verified Workflow for Election Teams

An operational deployment can therefore run from ingest to review in a defined order: preserve the source, detect text regions, track overlays, classify their function, extract values, attach election entities, and expose uncertain cases for checking. Searches can ask for every seat-total badge after 8 p.m., every appearance of a named candidate, or every replay carrying the same results panel. Quality monitoring flags unreadable frames and dropped overlays, while duplicate detection identifies repeated packages across channels.

Retrieval from the open web needs the same discipline. A page promoting online casino games guide might be pulled into a broad multimedia crawl because it contains motion graphics, percentages and calls to action. It has no electoral provenance, so proximity to election terms or visual similarity cannot make it part of a count. Keeping source type, collection reason and publication context in every record prevents unrelated promotional material from contaminating searches built around Australian election coverage.

The final quality check is editorial as much as technical. Reviewers verify that projections, declared totals and analysis panels are labelled correctly, that abbreviations and decimal separators are preserved, and that a candidate name is not inferred from a face alone. The practical takeaway is to treat every statistical overlay as a timestamped claim: retain the original frame, extracted value, graphic type, position and confidence, then publish only the records whose provenance and classification are clear enough to defend.