How ReCAP identifies and tags scenes with on-screen charts and graphs

Broadcast graphics have moved far beyond the static weather maps of the past. Today's news bulletins, business programs and weekend sport reviews routinely layer live bar charts, animated line graphs, electoral swingometers and pandemic curve trackers directly over the picture. For a viewer in Sydney, Melbourne or Perth, recognising these visual aids is second nature; for a software system analysing the same footage in real time, it remains one of the hardest perception problems in modern media engineering. The ReCAP project is tackling that problem head on, building automated detectors that recognise when a chart or graph appears, classify what kind it is, and tag the surrounding scene so that downstream tools can search, summarise or repurpose the footage.

This work matters because the volume of broadcast-quality video being produced and archived across Australia and Europe continues to climb. A single evening news bulletin can contain dozens of graphical inserts, each of which encodes information that viewers expect to find again later when they search an archive. Without reliable metadata, those moments disappear into the long tail of unread tape. ReCAP's approach treats on-screen charts and graphs as first-class citizens in the metadata schema, alongside faces, logos and quality metrics, and gives production teams a way to retrieve exactly the frame they need.

The technical goal is not merely to detect that something graphical is on screen, but to label what type of chart it is, which axis it uses, whether it is animating, and how long it remains visible. That level of granularity is what turns raw detection into genuinely useful tagging for media asset management, compliance monitoring and live editorial workflows. The rest of this article walks through how that pipeline is built, where it has been tested, and what production teams should keep in mind when adopting it.

Why charts and graphs are uniquely difficult to detect

Charts and graphs look nothing like the objects most computer vision systems were originally trained on. A bar chart is an arrangement of rectangles drawn over a coloured background, often partially transparent, frequently scrolling horizontally as new data points arrive. A pie chart can rotate. A line graph might pulse in time with a voice-over reading percentages. None of these patterns resemble a person, a building or a car, so generic object detectors trained on ImageNet-style datasets consistently miss them or mislabel them as generic screen content.

The difficulty grows when broadcasters combine charts with other visual elements. Australian election coverage routinely overlays a swingometer on top of a candidate's face, then animates a trend line across the lower third of the frame. That single frame contains at least four distinct regions of interest, and a naive detector will see only one. ReCAP's pipeline therefore treats chart detection as a region-segmentation problem rather than a single-label classification problem, scanning each frame for areas where pixel distributions, edge density and colour gradients deviate from the surrounding scene.

There is also the question of motion. Many financial bulletins use animated graphs that draw themselves line by line, while weather segments spin in satellite imagery behind a forecast chart. The system must recognise the chart both in its partially drawn and fully drawn states. ReCAP addresses this by sampling frames at multiple points in the animation cycle and comparing intermediate states against a learned model of chart skeleton shapes, which lets it tag the graphic even before the final data point has been plotted.

Inside the ReCAP detection pipeline

The first stage of the pipeline is a low-level frame analyser that runs on every incoming video frame and extracts candidate regions containing high-contrast geometric shapes. This is essentially a fast filter that throws away ninety-five percent of the picture before any heavier machine learning kicks in, which matters for live broadcast where latency budgets are measured in milliseconds. The filter looks for clusters of straight edges, regular spacing between vertical or horizontal bars, and the kind of axis tick marks that almost never appear in natural scenes.

Regions that pass the filter are handed to a convolutional neural network trained specifically on synthetic and real-world chart imagery. The network outputs both a class label, covering bar, line, pie, area, scatter and table formats, and a confidence score. Because the network has been trained on thousands of examples drawn from European and Australian broadcasters, it has learned to recognise the visual idioms used by each network's in-house graphics team, from the Seven Network's red-and-blue colour palette to the SBS News infographic style.

Once a chart is classified, a separate module reads its semantic content where possible. For bar and line charts, optical character recognition is run on axis labels and numeric callouts, so the system can produce tags such as unemployment rate, quarterly trend, percentage change. The result is a structured metadata record that travels with the video file and becomes searchable in any compliant media asset management system. The ReCAP consortium partners have documented this end-to-end flow as part of their public technical references on the consortium page.

Training data and the role of synthetic charts

Gathering enough real-world broadcast footage with accurately labelled charts would be impractical for a research project, so ReCAP has invested heavily in synthetic data generation. A rendering engine produces thousands of chart variants using the same fonts, palettes and animation styles observed in real bulletins. These synthetic charts are then composited onto real broadcast backgrounds, with realistic lighting, motion blur and on-screen text overlays to mimic production conditions.

The synthetic data is paired with a smaller, carefully curated set of real frames harvested from public broadcasts. Annotators in Melbourne, Madrid and Helsinki have labelled more than thirty thousand frames by hand, marking chart boundaries, axis positions and the type of each graphic. The two datasets are mixed during training, with the synthetic examples teaching the network structural variety and the real examples teaching it to handle broadcast noise, compression artefacts and the occasional ticker tape scrolling across the bottom of the screen.

A surprising benefit of this dual approach has been transferability. The model trained primarily on European financial bulletins generalises to Australian breakfast television with very little fine-tuning, because the underlying visual grammar of a bar chart is largely the same regardless of which network produced it. The team has published a short technical note on the data generation pipeline, which has been picked up by local research groups at RMIT and the University of Technology Sydney who are exploring adjacent problems in accessible broadcasting.

Working with Australian broadcast conditions

Australian broadcasters operate under conditions that occasionally catch overseas models off guard. The country spans three active time zones, with AEST, ACST and AWST all appearing in a single national bulletin when regional opt-outs are included. Chart animations are often synchronised to local time stamps, and the system has to recognise that a graph showing a Sydney time is logically the same chart as one showing a Perth time a few hours later. ReCAP's metadata layer captures the local time string directly from the on-screen text, leaving timezone normalisation to the downstream application.

The Australian Communications and Media Authority also imposes content rules that affect how charts are presented, particularly around electoral advertising and financial advice segments. Charts used in those contexts must carry specific disclaimers, and a tagging system that can flag the presence of a chart is also a useful audit tool for compliance teams checking whether the required text is on screen. ReCAP does not interpret compliance on its own, but its scene tags give compliance officers a fast way to jump to the relevant frames rather than scrubbing through an entire bulletin.

Locally produced graphics from networks such as ABC News, Nine and Ten also rely on distinctive lower-third templates that frequently overlap with chart elements. The detector handles this by treating overlap as a feature rather than an error, returning multiple tags for the same frame where appropriate. A single frame might therefore be tagged as bar chart, lower third and political party logo simultaneously, which mirrors how a human editor would describe the scene.

Accuracy, benchmarking and human review

Like any perception system, ReCAP's chart detector is not perfect. On a held-out test set of one thousand real broadcast frames drawn from Australian and European sources, the model achieves a mean average precision above 0.9 for chart presence detection and a class-level F1 score above 0.85 across the five most common chart types. Performance drops noticeably for pie charts smaller than ten percent of frame area and for hand-drawn illustrations that approximate but do not strictly follow chart conventions, which are common in editorial commentary segments.

To handle the inevitable errors, ReCAP ships with a lightweight review interface that lets an operator accept, reject or correct the automatic tags. In production, this is usually a junior production assistant who skims a daily bulletin in under five minutes, fixing any mislabelled frames before the metadata is committed to the archive. The corrections flow back into the project's training set, allowing the model to improve continuously.

The team has been careful not to oversell the technology. Charts remain a notoriously slippery category, and the project partners regularly remind integrators that automatic tags should be treated as a search accelerator rather than a replacement for editorial review. That honest framing has helped the consortium secure pilots with several major European broadcasters and at least one Australian public-service outlet, where the focus is on retrieval speed rather than autonomous decision making.

Practical guidance for production teams

Teams preparing to integrate ReCAP's chart tagging into an existing workflow should consider a handful of practical points before turning the system on across a full archive.

The next concrete step for any team interested in seeing the system in action is to request a live demonstration through the project website, where partners can run a recent bulletin through the detector and return a tagged sample within forty-eight hours. That kind of low-commitment trial is the fastest way to judge whether the accuracy figures hold up against your own footage, and it gives the ReCAP team the feedback they need to keep refining the model against real Australian broadcast material.