Automated detection of stage lighting changes in concert recordings

ReCAP, an EU-funded research initiative focused on real-time content analysis and processing, develops automated tools that scan broadcast-quality video for meaningful visual events. Its toolkit already flags face appearances, recognises logos, finds duplicate segments, and tracks camera quality issues across long recordings. The latest work targets an event type that has long frustrated production engineers: the rapid, scripted lighting cues that define a live concert broadcast.

In an arena setting, lighting designers trigger hundreds of colour and intensity changes during a single set. Beam positions snap across the stage, strobes pop at beat markers, and quiet ballads fade into deep ambers as the song moves into its bridge. To the human eye, these transitions are deliberate art. To a video analysis algorithm, they look like chaos. Standard shot boundary detection, built around camera cuts and editor-driven transitions, treats every flash as a potential scene change and floods the resulting metadata with false positives.

The ReCAP approach separates lighting cue events from camera transitions by measuring colour distribution and luminance drift frame by frame. When a designer shifts the rig from cool white to saturated magenta, the algorithm records the change, logs the timecode, and labels the colour profile. Editors can then search the recording later for any cue that matches a target palette or intensity band, and pull those moments straight into a rough cut without scrubbing through hours of footage.

This article explains how automated lighting detection works within the ReCAP analysis pipeline, where it sits alongside related tools for face recognition, logo detection, and camera shake logging. It also explores why this matters for the Australian market specifically, where concert recordings form a regular part of public broadcaster schedules and where local festivals generate volumes of footage that have historically been difficult to catalogue through manual effort.

Why concert footage breaks traditional scene detection

Conventional scene change detectors were built for scripted television. A drama moves cleanly from one setup to the next, each transition handled by an editor who follows a written shot list. News footage changes when a story breaks. Even live sports coverage follows predictable camera logic, with cuts aligned to replay boundaries or graphic overlays.

A concert breaks all three patterns in a single frame. The camera operator might hold a wide shot across an entire song while the lighting rig cycles through twelve different states in the background. From the perspective of a tool expecting human-driven transitions, those twelve lighting shifts look like twelve attempted cuts, none of which actually happened. Editors downstream end up with metadata that suggests the show was edited twelve times when in reality the operator never touched the cut button.

Manual tagging does not scale. A two-hour concert, archived for a music documentary or a streaming catalogue, can contain more than six hundred discrete lighting cues. Asking a junior assistant to catalogue each one is both expensive and error-prone, and the tags rarely reach a level of descriptive consistency that allows for later search. Production teams have been quietly accepting this loss for years because there has been no good alternative.

How ReCAP identifies lighting transitions

The technical core of the lighting detector is a rolling window of frame statistics gathered in HSV colour space. ReCAP samples each incoming frame, builds a normalised histogram for hue and value channels, and compares that histogram against a baseline established in the previous window. A lighting cue produces a sharp, sustained divergence between the two histograms, while ordinary stage motion, such as a performer walking across a spotlight, produces only short, locally contained variation.

The detector distinguishes between two event classes. A colour shift registers when the overall palette changes but overall luminance remains stable, for example when a song moves from a red wash to a blue wash without dimming. A dimming event registers when overall brightness drops or rises sharply, even if the hue stays similar. Both event types feed into a unified lighting log that downstream tools can subscribe to in real time. Tests on multi-camera recordings from European venues have shown precision above ninety per cent against human-labelled ground truth, with recall holding above eighty-five per cent even when rapid strobing is involved.

Because the detector runs as part of the broader ReCAP processing chain, it shares its metadata with other modules. A face recognition event that occurs during a cue tagged as deep blue can be cross-referenced with the lighting log to confirm the moment, and a logo recognition hit on a backdrop banner becomes much more useful when the lighting log confirms that the banner was fully illuminated at that timecode rather than half-washed out by a strobe.

Metadata output and editorial workflows

The lighting log leaves the ReCAP system in two complementary formats. A structured XML feed carries event records with start timecode, end timecode, detected colour profile, intensity band, and confidence score. A simplified subtitle track carries a human-readable description for every event, formatted to fit existing broadcast subtitle pipelines. Production teams can choose one or both depending on how their editing environment consumes metadata.

In practice, the structured feed is the more powerful tool. An editor searching for high-energy chorus moments during the second half of a headliner set can filter the log for warm-toned, high-intensity cues and be presented with a vertical list of timecodes, each linking straight to the relevant clip. This is the same search pattern that already powers sports highlight reels, now applied to a domain where the events are aesthetic rather than athletic.

The simplified subtitle feed solves a different problem. Australian public broadcasters, including the ABC and SBS, supply captions for online concert streams as part of standard accessibility practice. ReCAP lighting subtitles can ride alongside dialogue captions, giving visually impaired viewers an audio proxy for mood, for example "cool blue wash, dim" or "warm amber, full intensity". This is a small step in accessibility terms but a meaningful one for audiences who depend on descriptive audio.

Australian broadcast and festival context

Australia has a deep institutional relationship with concert broadcasting. The ABC has documented live performance for decades, from the long-running Studios sessions at the Sydney Opera House to more recent multi-cam productions from festivals such as Laneway and the longstanding Falls Festival stops at Lorne and Marion Bay. Melbourne's Hamer Hall and Brisbane's QPAC produce regular full-length concert recordings that move through digital archive pipelines before any air date is set.

For local production houses, the volume of footage is the central challenge. A multi-stage festival can produce more than forty hours of raw concert footage over a single weekend, and broadcasters may only need to assemble three or four hours of finished programming from that source. Anything that narrows the search space for editors saves both time and contract hours.

Local market realities also shape technical priorities. Australian broadcast standards still adhere to a mix of legacy Free TV Australia operational practice and post-2020 streaming requirements. ReCAP's structured metadata feeds slot neatly into both, since the XML output can be remapped into local asset management systems such as those used by the Australian Centre for the Moving Image in Melbourne or by Sydney-based post houses serving the international touring industry.

Camera jitter as a motion-domain counterpart

A separate body of work inside ReCAP focuses on camera operator behaviour rather than lighting behaviour, and it often appears in concert footage gathered from festival pit cameras or hand-held roving cameras. Handheld camera shake creates a different statistical signature from lighting cues, but the logging principle is the same. The companion project work on ReCAP for detecting and logging camera jitter and shake in handheld footage covers the motion-domain counterpart in detail, and both detectors share a common output schema so editors can search for "stable + cool tones" or "shaky + warm tones" using the same query language.

What the two modules demonstrate together is that ReCAP is moving toward an internal vocabulary of perceptual events rather than raw technical descriptors. Lighting cues and camera jitters are both translated into editorial language so producers can reason about mood and feel without ever looking at a histogram. The shared schema also simplifies tool maintenance, since a single dashboard can visualise both streams side by side when reviewing a recorded show.

Cross-domain transfer to other controlled lighting settings

The lighting detector was originally validated on concert material, but the same statistical core generalises to any controlled studio setting. Card-game studio recordings fall into this category, with baccarat studio workflows tested in early ReCAP pilots at Australian and Asian studios. The colour-distribution logic behaves identically in both domains, since a sharp, sustained palette shift flags a lighting event regardless of whether the subject on stage is a singer or a dealer at a card table.

Transferring the detector across domains requires only modest retraining. Operators annotate a small sample of representative frames from the new environment, the system adjusts its baseline thresholds, and detection continues without code changes. This is why ReCAP's design team split the analysis into a library of small statistical tests rather than one monolithic classifier. Each test can be retuned independently, and new domains can be onboarded with relatively small amounts of training material.

The generalisation also extends downward in production value. Lighting detection runs cleanly on smartphone footage shot by audience members at a Melbourne pub gig, as well as on broadcast-grade multi-cam from a Sydney headline show. Smaller and noisier inputs simply produce lower confidence scores, and downstream tools learn to weight those scores accordingly when surfacing results.

Practical pipeline integration and broader reach

For post-production, the most immediate benefit is the searchability of long concert masters. Editors preparing a documentary about an Australian artist's tour can search the archive for "purple wash, lead vocals" and pull every occurrence across multiple nights in a few seconds. For a broadcaster preparing a same-day highlights package, the ability to identify the most visually dramatic chorus cues within minutes of the show ending is the difference between catching the share cycle and missing it.

For archivist roles, lighting logs provide forensic value long after the initial broadcast. Decades from now, a researcher studying how a particular Australian artist used colour across a tour can query the archive and reconstruct the visual arc of every show, even if no human ever sat down to catalogue each cue by hand. Cultural institutions such as the National Film and Sound Archive of Australia, with its Canberra headquarters, are already exploring such datasets for research access. Quality monitoring is another quiet beneficiary. When a camera loses exposure in the middle of a song, the resulting metadata event often looks similar to a lighting transition, and ReCAP's cross-checking between modules helps distinguish the two, flagging genuine technical faults that would otherwise be hidden inside the noise of expected artistic choices.

Beyond traditional broadcast, the same change-detection principles are now being applied to other categories of live streamed video where graphical elements shift rapidly in a stable composition. Visual quality monitoring modules derived from ReCAP are being trialled on streaming interfaces for live gaming categories such as interactive casino streaming portals, where graphical elements transition rapidly during play. The histogram comparison logic that flags a lighting cue in a concert also flags a graphical transition in a live gaming broadcast, because both are fundamentally about detecting structured visual change in a stable composition.

This portability is one of the more useful outcomes of designing detection as a set of small, interchangeable statistical tests rather than as one monolithic classifier, and it is the direction in which the broader ReCAP roadmap is now moving.

A practical next step for any Australian production team working with concert material is to run a short pilot of the ReCAP lighting detector against an archived show, comparing the generated cue log against the existing editorial slates. The project team can be contacted through the website to arrange a pilot dataset and a short assessment window.