How ReCAP processes video carrying 3D stereoscopic metadata

Stereoscopic video is no longer a niche curiosity reserved for science museums in Melbourne or specialty cinema chains in Sydney. With the rise of immersive sports coverage, 3D-capable streaming services, and an Australian audience that has enthusiastically adopted VR headsets for both gaming and live events, broadcasters and production houses are routinely handling footage that carries depth information. This depth information is rarely present in the visible pixels; instead, it is embedded as metadata that travels alongside the video bitstream. When that metadata is stripped, misinterpreted, or mangled by automated processing tools, the viewing experience collapses into eye-straining double images or flat 2D with no warning at all.

The Real-time Content Analysis and Processing project, better known as ReCAP, was designed from the outset to read, interpret, and preserve the rich metadata embedded in modern video files. Whether the source is a multi-camera live feed from the Sydney Cricket Ground, a post-produced documentary from a Brisbane studio, or archived material being prepared for re-release, the framework treats stereoscopic flags, disparity maps, and depth cues as first-class citizens rather than optional extras. The result is a pipeline that media engineers can trust to keep the three-dimensional context intact from ingest to playout.

This article walks through the technical decisions behind ReCAP's handling of 3D stereoscopic metadata, the quality checks the system applies, and the practical implications for Australian media organisations working with immersive content. It also touches on related capabilities such as face recognition and duplicate detection, because metadata rarely exists in isolation within a modern asset management environment.

Reading the signal: how stereoscopic metadata enters the pipeline

Broadcast engineers in Adelaide and Perth are increasingly familiar with files that announce their 3D nature through container-level flags. The most common signals come from the MVC extension to H.264/AVC, the frame-packing arrangement SEI messages in HEVC, and the more recent VVC profiles that explicitly encode left and right view ordering. ReCAP's ingest module treats these signals as structured data, not as opaque bytes. Each file is parsed to determine whether it represents a side-by-side arrangement, a top-and-bottom layout, a frame-sequential stream, or a true two-view MVC encode.

Once the layout is identified, the system records it as an attribute on the asset record rather than rewriting the bitstream prematurely. This matters because Australian broadcasters often need to maintain a single canonical master while generating multiple downstream deliverables: a stereo version for Foxtel's experimental channels, a mono downmix for catch-up services, and a per-eye extract for editorial review. By preserving the original layout intent, ReCAP lets downstream tools reason about the content without committing to a single delivery format too early in the chain.

The parser is also defensive. Older camera systems from manufacturers that no longer exist still produce files with non-standard stereoscopic flags, and some legacy editing suites inject proprietary markers that can confuse downstream decoders. ReCAP maintains a profile database that maps these quirks to canonical interpretations, logging a confidence score alongside the parsed metadata so that downstream consumers in newsrooms from Canberra to Hobart can decide whether to trust the automatic interpretation or flag the asset for manual review.

Quality monitoring that respects depth

Video quality monitoring is a core deliverable of the ReCAP project page, but the project's quality assurance routines had to be extended to handle stereoscopic content meaningfully. Traditional blockiness, blur, and noise metrics are computed per-view, then aggregated with explicit awareness that the two views must remain synchronised. A forty-millisecond drift that nobody would notice on a single 2D feed becomes a nausea-inducing artefact on a stereo broadcast, so ReCAP's temporal alignment checks run with sub-frame precision across both views simultaneously.

The system also evaluates perceptual depth consistency. If the left view claims a foreground subject at one depth and the right view disagrees, ReCAP flags the asset and provides a short clip excerpt to help the operator diagnose the issue. This kind of cross-view check has proven valuable for live productions at venues such as the Melbourne Cricket Ground, where multicamera setups occasionally fall out of calibration during a long sporting event. Operators receive an alert long before the mismatch reaches air, and the relevant segment can be re-rendered from the original camera feeds.

For archival workflows, ReCAP runs a slower, offline variant of these checks. The archived material is processed against the project's quality baselines, and any stereoscopic-specific degradation such as colour drift between views, resolution mismatch, or compression artefacts that affect depth perception is logged in a structured report. Libraries that hold cultural material, such as the National Film and Sound Archive in Canberra, benefit from these reports when they plan digitisation projects that need to preserve depth information for tomorrow's display technologies.

Metadata preservation through transcoding and archival

Transcoding is where stereoscopic metadata is most often lost. A generic transcoder that does not understand the input layout will either flatten the content to 2D or produce a single-view output that looks superficially correct but has discarded the second view entirely. ReCAP addresses this by treating transcoding as a metadata-aware operation. The pipeline carries the stereoscopic layout flag, the per-view resolution, and any disparity or depth metadata through every transformation step, rewriting them into the new file with the correct container syntax.

When the target format does not support a given stereoscopic feature — for example, when producing a legacy MPEG-2 deliverable for a regional affiliate — the system logs the lossy conversion and offers a sidecar metadata file that records the original 3D structure. This sidecar approach mirrors practices already familiar to Australian post-production houses that work on long-form drama series, where continuity notes are often delivered alongside the broadcast master. Operators can later reconstruct the original stereo intent if the deliverable is ever re-purposed for a 3D-capable platform.

Archival storage is treated similarly. ReCAP can write to object stores that preserve the full set of metadata fields, including the ones that current players may ignore. The project's consortium partners have validated this against long-term preservation systems used by European broadcasters, and the same patterns translate well to Australian cloud deployments. The principle is simple: do not throw away information that might become valuable once the next generation of immersive displays reaches consumers in Sydney's flagship electronics stores or regional shopping centres.

Faces, logos, and stereo-aware recognition

ReCAP's face recognition and logo detection modules were not originally designed with stereoscopic content in mind, but the project's modular architecture made it straightforward to extend them. When a stereo asset is identified at ingest, both views are processed, and the recognition results are merged using a depth-aware voting scheme. If the same person appears in both views, as they should in a properly captured stereo pair, the recognition confidence is reinforced. If a face appears in only one view, perhaps because of occlusion by a foreground object, the system records that asymmetry as part of the metadata.

This dual-view processing has practical benefits for live sports coverage in Australia. Sponsorship logos painted on the field of play at Brisbane's Suncorp Stadium, for instance, may be partially occluded by players in the left view but fully visible in the right view at certain moments. Combining the two views yields a more complete picture of brand exposure than either view alone, which is useful for rights holders and broadcasters who monetise that visibility. The duplicate detection module, meanwhile, treats left and right views as related but distinct, so a near-duplicate search will not falsely flag a stereo pair as a copy of itself.

The project has published technical notes on how these stereo-aware recognisers integrate with broader media asset management systems. Operators do not need to learn a new interface; the metadata simply appears in the standard fields they already query. For organisations evaluating similar automation practices, the consortium has also pointed to adjacent community resources, including discussion threads on tools for custom sticker reaction bots, which illustrate how metadata-driven automation is becoming routine in adjacent industries.

Operational integration and practical lessons

Rolling ReCAP into an existing broadcast workflow is rarely a matter of dropping in a new box. The project's reference implementation exposes a documented API and a set of command-line tools, but most Australian deployments to date have wrapped those interfaces in their own orchestration layers. A typical installation at a regional broadcaster might run ReCAP as a microservice that watches an incoming folder, processes new files, and writes enriched metadata back to the existing asset management database. The stereoscopic handling is invisible to the operator unless they specifically inspect the metadata fields.

A few practical lessons have emerged from early deployments. First, always validate that the source file's declared layout matches what the operator believes they shot. Second, run a short stereo quality probe on a representative sample before committing a large batch to transcoding. Third, retain at least one version of the original master with its full metadata, even if the immediate deliverable is a flattened 2D file. Australian production teams have found that these habits save significant time when a delivery is repurposed months or years later.

The project consortium continues to gather feedback from operational sites and incorporates it into successive software releases. The blog on responsible workflows carries related commentary on metadata hygiene and automated quality checks, and the same principles apply directly to stereoscopic content. The message is consistent across the project's communications: treat metadata as a first-class part of the asset, not as an afterthought to be cleaned up after the video has been delivered.

Key capabilities for stereo-aware processing

What to watch for in a stereo pipeline

Stereo video will not replace conventional 2D broadcasting in Australia any time soon, but it has earned a permanent place in the toolkit of producers who serve sports fans, cultural institutions, and immersive entertainment audiences. The defining lesson from ReCAP's work on stereoscopic metadata is straightforward and worth holding onto. Depth information is metadata, metadata is signal, and signal deserves the same care as the pixels it describes. Australian media organisations that build this discipline into their ingest, monitoring, and archival routines today will find themselves ready for whatever immersive formats the next decade brings.