How ReCAP classifies video segments by lighting and color temperature

Real-time video analysis has moved well beyond simple face detection. Modern broadcast operations depend on granular, frame-level understanding of what is actually on screen, and lighting is one of the most telling signals of all. The ReCAP project, an EU-funded research initiative on broadcast-quality content analysis, treats illumination and color temperature as first-class metadata. By tagging every segment of footage with these characteristics, the system gives editors, archivists, and quality engineers a far richer view of their assets than a generic timestamp ever could. The result is a searchable map of how each clip was lit, ready to be queried without anyone needing to scrub through footage by hand.

For Australian media organisations, this kind of detail has practical weight. A broadcaster in Sydney might receive rushes from a Brisbane news crew at noon, a Melbourne drama unit under tungsten, and a Perth remote shoot at dusk, all in the same working day. Being able to group those clips by their dominant lighting profile changes how production teams search, re-use, and restore material, especially when deadlines are tight and the colour grading suite is already booked. Lighting classification also affects rights management, advertising placement, and continuity checks, three areas where the Australian industry has invested heavily in recent years.

The challenge of lighting-aware metadata in Australian production

Australia's geography forces unusual demands on camera crews. In a single afternoon, a cinematographer might move from the soft coastal haze over Bondi Beach to the high-contrast noon glare of a Pilbara mine site, then to the warm tungsten interior of a heritage pub in Hobart. That range of ambient conditions is wider than what many European or North American production pipelines were built for, and it places a real burden on post-production teams who must reconcile wildly different source material into a coherent programme. The country's reputation for harsh ultraviolet light, deep coastal shadows, and rapidly shifting weather patterns makes every lighting decision on set feel consequential.

ReCAP approaches the problem by treating lighting and colour temperature as searchable properties rather than as accidental artefacts. The analysis pipeline reads luminance distribution, white balance drift, and spectral characteristics directly from the video stream, then assigns each segment to a descriptive lighting class. That classification is portable, machine-readable, and consistent regardless of who shot the footage or which camera body captured it. Operators do not need to manually tag assets or trust the often-inconsistent metadata written by camera operators in the rush of a live event.

For Australian broadcasters juggling time-zone sensitive live feeds across AEST, ACST, and AWST, this consistency is more than a convenience. It allows a control room in Ultimo to quickly identify whether a regional cut was shot under daylight fluorescents, mixed indoor lighting, or controlled studio LEDs, and to make rapid automated decisions about downstream processing. When a feed from regional WA needs to be dropped into a national bulletin at short notice, knowing its lighting profile in advance avoids footage that suddenly looks out of place next to studio-shot material.

Inside ReCAP's colour temperature detection workflow

At the heart of the ReCAP pipeline is a hybrid model that blends computer vision heuristics with learned feature extraction. The system evaluates frames at multiple sampling rates, looking for stable lighting signatures that persist long enough to be meaningful but shift when the camera relocates or the sun breaks through cloud cover. From those signatures it derives a correlated colour temperature estimate, expressed in kelvin and grouped into familiar categories. The model was trained against a curated dataset that includes deliberately unbalanced lighting scenarios, because real broadcast material rarely arrives in a tidy package.

The pipeline handles the common scenarios that Australian crews encounter:

Each segment is tagged with both the numerical estimate and the categorical class, which gives technical operators the precision they need while keeping editorial search interfaces intuitive. The classification respects cuts and fades, so a single piece of footage containing two distinct lighting environments receives two distinct tags, with a transition marker recorded between them. Boundary awareness matters in long-form content such as cricket coverage, where innings breaks and lighting rig changes both happen within a single broadcast day.

Real-time tagging for live broadcasts and fast-turnaround news

Live workflows leave no time for manual review, and that is exactly where the ReCAP design philosophy pays its way. The system is engineered to operate within a broadcast-quality latency budget, producing lighting classifications as the video flows through the processing chain. News producers in Adelaide or Darwin dealing with sudden bushfire coverage, for instance, can rely on the system to flag whether incoming citizen footage was shot at twilight under smoke-tinted skies, an important cue for how the material should be colour-corrected and presented on air. Such footage often arrives unprocessed, and a lighting classification layer helps journalists decide at a glance whether a clip is worth chasing down for rights clearance.

The same capability supports sports production, where lighting profiles shift dramatically across a match. Night fixtures under stadium LEDs, twilight sessions that begin in daylight and finish under floodlights, and day games at the SCG or the Gabba all produce different spectral signatures. ReCAP's segment-level tagging means that highlight reels, sponsor reels, and archival cuts can later be filtered to find every shot captured during the floodlit portion of a BBL cricket match, without anyone needing to scrub through hours of footage manually. This is useful when broadcasters want to reuse material from previous seasons to build season-launch packages.

Handling transitions, mixed light, and graded material

Real footage rarely sits in a single lighting category for long. A camera following a politician from an outdoor press conference into a hotel lobby will pass through at least three distinct lighting regimes in a few seconds, and a drama scene may use graded footage deliberately shot under warm tungsten before being pushed cooler for narrative effect. ReCAP's classifier is built to recognise these transitions explicitly rather than averaging them into a single misleading tag, and the system flags genuinely mixed material as such.

Graded footage presents a different challenge, since colour correction can shift original daylight material into a warm tungsten-looking scene. ReCAP mitigates this through feature channels that capture lighting characteristics more robust to grading, and exposes a graded-versus-raw indicator that operators can use when needed. For Australian re-mastering projects working with older drama series, this distinction is particularly valuable because it helps separate material that needs colour restoration from material that simply looks aged due to deliberate grading choices.

Applications across Australian media asset management

Australia's media sector is unusually concentrated, with a handful of major broadcasters and production houses serving a vast geography. Most of those organisations operate substantial MAM libraries, often running into the hundreds of thousands of hours of stored content. ReCAP's lighting and colour temperature metadata slots directly into those archives, extending existing search taxonomies without requiring wholesale replacement. For organisations that have invested in digitisation programmes around archival newsreel material, this added metadata layer is essentially free intelligence on assets that already exist.

Practical uses that have emerged from the project's trials include:

The last of these is particularly relevant for archival work. Many historical Australian broadcasts were shot under now-obsolete lighting standards, and a classification layer makes it possible to identify those segments at scale rather than relying on slow, manual inspection. Researchers and documentary producers working with old footage from Channel Nine or the ABC can quickly locate material captured under specific lighting conditions, such as early videotape productions lit primarily with fluorescent tubes, which often need particular care during restoration.

Integration with existing broadcast pipelines

Adoption hinges on whether the technology slots into established workflows without forcing operators to learn new tools. ReCAP exposes its classifications through standard metadata schemas, allowing downstream systems to consume lighting tags without modification. For facilities running Vizrt, Dalet, or Avid iNEWS environments, the output appears as just another metadata field, one that existing search and scripting tools can already address. This matters in particular for Australian operations that have standardised on a small number of vendor ecosystems and cannot easily accommodate proprietary metadata formats.

The project has paid particular attention to interoperability with QC systems. Lighting and colour temperature classifications complement traditional checks for blockiness, dropouts, and audio levels, giving quality engineers a richer diagnostic picture. When a clip is flagged for review, the operator can immediately see whether the issue might stem from misjudged white balance under mixed light, or from genuine compression artefacts, before a costly re-grade. This kind of triage saves considerable time during fast-turnaround news cycles.

For Australian production houses that contract work across state borders, this interoperability also reduces friction. A post-production supervisor in Brisbane can ingest a Sydney-delivered master, see its lighting profile at a glance, and decide whether the material needs additional balancing before it joins a national release. Independent producers delivering content to multiple networks, who often face differing technical specifications, can confirm in advance that their lighting handling will meet each broadcaster's expectations.

Performance, accuracy, and what comes next

Benchmarking against curated test sets has shown that ReCAP's lighting classifier reaches high agreement with human graders across the lighting categories most relevant to broadcast work. Colour temperature estimates are reported with a confidence interval, so downstream systems can choose how strictly to act on each tag. Misclassifications tend to cluster around genuine edge cases, such as dusk transitions or heavily graded archival material, which are precisely the scenarios where a confidence flag adds value. The consortium has been careful to publish both successes and limitations in its technical write-ups.

Ongoing research is focused on tighter temporal segmentation, faster adaptation to camera-specific colour profiles, and extension of lighting classification to HDR material as Australian broadcasters begin wider HDR trials. Work is also underway on linking lighting classification to camera identification, letting post-production teams apply camera-specific correction profiles automatically once a body is recognised. The consortium regularly publishes technical updates and field notes, and the project blog carries detailed write-ups of recent evaluations, including Australian-relevant test material.

For teams evaluating the technology, the core objectives page outlines the full analytical pipeline, from face and logo recognition to duplicate detection, and shows where lighting classification fits. Reviewing that document alongside trial footage quickly reveals whether the metadata depth matches a given production environment.

Australian broadcasters, post houses, and archive operators interested in piloting the lighting classification module in their own pipelines can request a trial dataset from the consortium through the contact form on the project site, with evaluation packs being prepared for the next round of partner onboarding in the southern spring.