How ReCAP Processes Video With Mixed Frame Rates From User-Generated Content
Footage uploaded by viewers rarely arrives in a clean, uniform state. A clip filmed at a Sydney news scene might roll at 29.97 progressive frames per second, while a tourist video from a Melbourne phone goes variable between 24 and 60 fps, and a home movie shot on an older camcorder hovers around 25 fps interlaced. Editors at Australian broadcasters routinely receive this kind of mixed-source material within minutes of an event and expect their tools to handle it without complaint.
ReCAP, the EU-funded Real-time Content Analysis and Processing initiative, was designed to operate in such conditions. Its pipeline ingests user-generated content, normalises frame timing, extracts metadata, monitors quality, recognises faces and logos, and flags duplicates on the fly. The project work plan lays out how each module cooperates, and the sections below trace what actually happens when frames arrive at inconsistent rates.
Why Mixed Frame Rates Challenge Traditional Pipelines
Traditional broadcast chains assume a fixed cadence. A studio gallery in Sydney expects 25 fps interlaced, a streaming workflow in Melbourne could target 50 fps progressive, and a social media clip on an Australian creator's phone can fall to 24 fps under low light. When these streams meet in the same queue, naive processing produces juddering motion, dropped audio cues, and broken timestamps.
The variability is more than numerical. Some phones record variable frame rate, dropping frames when the sensor overheats. Action cameras frequently switch between 30, 60, and 120 fps depending on resolution. Older digital cameras locked at 25 fps interlaced still appear in viewer-submitted archives. Under battery strain or temperature shifts, sensors throttle and add jitter that no operator can predict.
| Frame Rate Family | Typical Source | Strengths | ReCAP Handling |
|---|---|---|---|
| 23.976 / 24 fps progressive | Cinema phones, DSLRs | Cinematic feel | Cadence-locked to broadcast reference |
| 25 fps interlaced | Legacy broadcast cameras, PAL phones | Compatible with older edit bays | Field-based frame doubling |
| 29.97 / 30 fps progressive | Modern handsets | Smooth motion for online | Direct mapping to PTP timestamps |
| 50 / 60 fps progressive | Sports phones, action cameras | Crisp slow-motion | Decimated for archives, retained for highlights |
| Variable (VFR) | Dashcams, low-end handsets | Flexible capture | Re-timed using embedded timestamps |
Each row represents a real category of contributor footage that newsrooms and production houses have flagged as problematic. The right-hand column shows the first-pass strategy the system applies before deeper modules take over. Detailed cadence mapping notes are published through the communications assets portal so partners can audit the strategy against their own contribution feeds.
Frame Rate Normalisation As The First Stage
The first module in the chain is temporal normalisation. ReCAP examines the container metadata and the bitstream itself to determine whether the clip is constant frame rate, variable frame rate, or field-based. For Australian broadcasters, this step matters because local ACMA technical standards expect timecodes aligned to a recognised reference, even when the underlying footage was never shot that way.
When the detector identifies a constant rate, the pipeline maps each frame to a Precision Time Protocol timestamp using a nearest-neighbour rule, so an incoming 30 fps stream aligns cleanly with a studio 25 fps reference through frame blending. For variable rate sources, ReCAP re-times the stream against embedded timecodes, interpolating between samples rather than guessing. For interlaced sources, fields are merged through motion-adaptive deinterlacing, preserving the original motion intent while producing progressive frames downstream.
Metadata writers then attach the chosen cadence to the clip header, so downstream tools can trust the timing. A clip that arrived as 59.94 fps variable on a contributor's device leaves the normalisation stage as a stable 25 fps progressive stream with locked timecodes, ready for quality and recognition modules.
Metadata Extraction Across Variable Timing
Once cadence is stable, metadata extraction begins. The same scene analysed at 24 fps and at 60 fps yields very different feature densities, so ReCAP adjusts its extraction profile to the effective rate. Sparse streams receive wider temporal windows, dense streams receive narrower ones. The result is comparable output regardless of input.
Location tags benefit from this approach. A clip geotagged at a coastal landmark carries the same confidence whether it was sampled at 24 or 60 fps, because the system anchors metadata to the frame nearest the GPS fix. Speech-to-text runs against the re-timed audio, so Australian accents transcribe consistently across the rate spectrum. Object detection overlays on production fields, so a beach rescue scene yields the same tags whether the footage is 30 or 50 fps.
The extractor also writes a frame-rate provenance field, a structured line of metadata that records the original cadence and the chosen normal. Editors downstream can see, at a glance, whether a clip was originally 24 fps cinema-style or 60 fps sports-style, which shapes editorial decisions on slow-motion and colour grading.
Quality Assessment For Consumer-Captured Footage
User-generated content is noisy, dark, shaky, and sometimes partly occluded. ReCAP's quality assessment module runs after temporal normalisation, so it scores a stream that already has stable timing. This avoids the trap of penalising a low-light phone clip simply because it dropped frames under sensor strain.
The assessment covers sharpness, exposure, colour balance, and audio clarity. A shaky clip scored at 60 fps gets a different sharpness profile than the same scene filmed at 24 fps, because motion distribution differs. The module compensates by using cadence-aware models trained across frame rates, and feeds a normalised score into the editorial dashboard.
Quality metadata is also written into the clip sidecar, alongside the provenance and recognition overlays, allowing moderation tools downstream to filter out clips that fall below a configurable threshold without losing their timestamp integrity.
Face and Logo Recognition On Inconsistent Streams
Recognition models are trained on frame sequences, so inconsistent timing throws off their assumptions. ReCAP's recognition pipeline uses the normalised stream from earlier stages, then re-injects cadence context where it matters. Face tracks benefit from longer observation windows on 25 fps sources, while logo recognition benefits from shorter, sharper sampling on 60 fps sources.
A crowd scene captured on a phone at 30 fps, and the same scene captured by a journalist at 60 fps, both yield face tracks of comparable reliability. The system uses the normalised stream for the core recognition pass, then overlays cadence context for refinement. Logos on jerseys and signage are matched against a reference library that includes Australian brands, broadcasters, and sporting bodies.
Duplicate recognition at this stage catches near-identical faces captured at different frame rates, a common problem when two Sydney newsrooms cover the same press conference and each receives a different-quality clip. The system flags these for the duplicate module downstream.
Duplicate Detection In Cross-Frame-Rate Archives
Duplicate detection is where cadence variability hurts most. A fingerprint computed on a 24 fps stream differs from one on a 60 fps stream, even when the content is identical. ReCAP computes fingerprints on the normalised output, so cadence becomes a controlled variable rather than a confounding one.
The module then compares fingerprints across the corpus, including legacy archives at standard rates, modern streams at variable rates, and historic footage in older formats. A clip sent at 30 fps matches a clip already in the archive at 25 fps, with the system flagging both for editor review.
This capability is especially valuable for production houses working under tight turnaround windows. When the same event arrives twice from different contributors at different frame rates, the duplicate flag prevents accidental double-counting and protects editorial integrity across a bulletin cycle.
Real-Time Delivery For Australian Broadcasters
The processed output reaches Australian broadcasters in real time through standard contribution feeds. ReCAP's pipeline runs at contribution speed, so a clip uploaded from a regional studio over NBN can be processed and returned before the next news bulletin cycle.
For broadcasters serving Australian audiences, latency drives editorial timing. A Sydney studio expects a clip returned within minutes, regardless of original frame rate, so its breakfast show runs to schedule. Regional broadcasters using satellite contribution benefit from the cadence normalisation, because the satellite hop introduces its own timing artefacts that the pipeline absorbs.
Editorial coordinators sometimes rely on batch messaging tools to alert contributors when clips are ready for re-use, and ReCAP's webhook layer supports such integrations out of the box.
The processed stream arrives at the broadcaster with stable timing, rich metadata, quality scores, recognition overlays, and duplicate flags. From upload to broadcast, the system treats mixed frame rates as routine rather than exceptional.
A practical starting point for any Australian partner is to feed a one-minute mixed-frame-rate clip from a recent local event into the project's demonstration pipeline, then compare the returned metadata against an existing broadcast reference clip from the same scene to verify cadence alignment, quality scoring, and duplicate behaviour in a single pass.