How ReCAP handles video with timecode discontinuities
Video analysis depends on a reliable relationship between frames, timestamps, and the events shown on screen. In a perfect file, timecode advances steadily from one frame to the next. Real broadcast material is rarely so simple. A recording may contain a camera restart, a programme splice, a missing segment, a timestamp reset, or a switch between sources with different clocks.
These irregularities are known as timecode discontinuities. They can affect every downstream task: scene detection, speaker identification, logo recognition, quality monitoring, duplicate-content discovery, and the creation of searchable media metadata. If an analysis system treats every timestamp as continuous, it can assign the wrong duration to a shot or place an event at an incorrect position.
ReCAP addresses this problem as part of its broader work on real-time content analysis and processing. The ReCAP project develops methods for extracting meaningful information from broadcast-quality video while keeping the results useful for production, live transmission, and media asset management. Handling time correctly is essential to making those results trustworthy.
Why timecode discontinuities matter
A timecode discontinuity occurs when the apparent timeline changes unexpectedly. The change may be a forward jump, a backward jump, a reset to zero, or a gap caused by missing frames. A stream can also contain a discontinuity when two recordings are joined and their clocks do not align. In live production, switching between cameras or contribution feeds can create similar effects.
The visual content may remain perfectly understandable while the metadata becomes misleading. Suppose a face appears at 01:12:08:10, then the next frame is labelled 00:03:14:00 because a recorder restarted. A simple duration calculation would interpret the second frame as occurring before the first. A duplicate detector might split one programme into unrelated fragments, while a quality monitor could report a false freeze or a massive timing error.
Different media formats expose timing in different ways. Container timecodes, presentation timestamps, decoding timestamps, frame numbers, and embedded ancillary timecode may disagree. Variable-frame-rate material introduces another complication because elapsed time is not always represented by a fixed number of frames. Reliable processing therefore begins by treating timestamps as evidence that must be checked, rather than as unquestionable frame labels.
Detecting a change in the timeline
A robust pipeline examines the timestamp sequence as frames arrive. It compares the current presentation timestamp with the expected timestamp derived from the previous frame, the declared frame rate, and any known stream configuration. A small rounding difference is normal. A large positive or negative deviation indicates that the timeline may have changed.
The detection stage can use several signals together:
- A backward timestamp movement or reset
- A gap larger than the expected frame interval
- A sudden change in frame rate, resolution, or stream identifier
- An explicit discontinuity marker in the container or transport stream
- A change in embedded timecode that conflicts with decoded timing
- A missing, repeated, or corrupted frame sequence
The threshold should be adapted to the material. A 100-millisecond gap means something different in a 25-frame-per-second programme than in a low-frame-rate contribution feed. It is also important to distinguish an actual break from normal encoder behaviour. B-frames, packet reordering, and decoding delays can make decoding timestamps appear irregular without changing the presentation order.
For this reason, ReCAP-style processing benefits from separating decode order, presentation order, and editorial time. The decoder needs one ordering to reconstruct pictures, while an analyst needs another to place events on a usable media timeline. Keeping those concepts distinct reduces false alarms and makes later metadata easier to interpret.
Preserving both source and analysis time
Once a discontinuity is found, the system should not simply overwrite the original timecode. The source value may be important to an editor, broadcaster, archivist, or compliance team. Instead, a processing record can preserve the original timestamp and create a second, normalized analysis timeline.
The source timeline answers questions such as, “What timecode was present in the delivered material?” The normalized timeline answers, “Where does this frame occur within the continuous sequence used for analysis?” Both values can be stored with a segment identifier and a relationship between segments.
A practical representation might include:
| Field | Purpose | Example |
|---|---|---|
| Source timecode | Retains the timestamp carried by the media | 01:12:08:10 |
| Presentation timestamp | Orders the frame for playback | 1,872,340 ms |
| Normalized position | Provides a continuous analysis coordinate | 4,328,100 ms |
| Segment ID | Identifies the timeline region | Segment 03 |
| Discontinuity type | Describes the detected change | Backward reset |
| Confidence or evidence | Records why the boundary was created | Timestamp and stream reset |
The normalized position can be calculated by adding the duration of completed segments and the relative position within the current segment. If a new segment begins after a reset, the first frame in that segment receives a new local origin while its source timecode remains unchanged. This allows an event to be searched by source timecode and still plotted correctly on a continuous analysis view.
The choice between joining segments and keeping them separate depends on the workflow. Automated tagging may require a continuous coordinate for indexing. Broadcast compliance may need a clear boundary because the source clock changed. A media asset management system can store both approaches: one asset with several timed regions, or several linked sub-assets with their original timing preserved.
Coordinating analysis across a discontinuity
A timecode break should become a controlled state change in the processing pipeline. When the system detects a boundary, it can close the current analysis segment, flush pending frame data, create a new segment context, and continue processing. This avoids carrying assumptions from the old timeline into the new one.
Feature extraction needs special care at the boundary. A shot detector may need to discard a comparison window that spans the reset because the two neighbouring frames are not adjacent in real media time. A motion estimator should avoid interpreting the first frame after a gap as an extreme camera movement. Audio-video synchronisation checks should also restart or rebase their local measurements.
Recognition tasks can continue, but their temporal state may need to be scoped. A face tracker should not link a person across a long missing interval without supporting visual evidence. A logo tracker may retain a short-lived state across a small packet loss, yet reset after a recorder restart. The same principle applies to speaker activity, subtitles, captions, and object tracks: preserve identity only when the evidence and timing support continuity.
This is especially relevant to automated tagging. ReCAP’s work on speaker-based interview tagging illustrates why event boundaries and temporal metadata must remain aligned. If an interview segment is associated with the wrong time range after a reset, a correct speaker label can still become operationally useless. Segment-aware timestamps make the tag traceable to the right portion of the source.
Managing live and file-based workflows
In file-based analysis, the complete media object may be available before processing begins. The system can inspect container metadata, scan for timestamp anomalies, and build a preliminary timeline before running expensive recognition models. This first pass helps determine where processing contexts should start and stop.
Live streams require a different strategy. The system cannot wait for a complete file, and some discontinuities are announced only when they occur. A streaming processor therefore maintains a rolling timing model. It estimates the expected next timestamp, watches for explicit markers, and emits a boundary event as soon as the evidence is strong enough.
Low latency introduces a trade-off. If the threshold is too sensitive, ordinary jitter generates unnecessary segments. If it is too tolerant, the system may attach metadata to the wrong timeline. A useful design can mark a boundary as provisional, process frames under a new local clock, and revise the event when later packets confirm the change. Downstream consumers should receive the boundary, its type, and its confidence rather than only a silent timestamp correction.
Clock alignment becomes more complex when multiple feeds are analysed together. Camera feeds, programme output, clean feeds, and replay sources may use different origins or drift over time. A common wall-clock reference can help, but it should not replace the source timecode. Correlation can also use visual cuts, audio peaks, captions, or known programme events to verify that streams are aligned.
Quality control around discontinuities
Timecode handling is closely tied to video quality assessment. A break in timestamps may be a harmless editorial splice, or it may indicate packet loss, a recorder fault, dropped frames, or an encoder restart. The analysis system should therefore classify the event instead of reporting every discontinuity as a quality failure.
Useful diagnostics include the size and direction of the timestamp change, the number of absent frames, decoder warnings, audio continuity, and whether the picture freezes or changes normally at the boundary. A clean cut with a documented timecode reset can be treated differently from a stream that loses several seconds and repeats the last image.
Reports should expose the distinction clearly. An operator may need to know that a programme contains three source-timecode segments, while a media manager may care that the normalized duration is 52 minutes and 14 seconds. A quality dashboard can show a timing event with severity, evidence, and affected range. Search and annotation tools can then use normalized time without hiding the original production context.
Validation should include difficult material rather than relying on continuous test clips. Test collections can contain backward jumps, large gaps, repeated frames, variable frame rates, concatenated files, transport-stream discontinuity indicators, and mismatched audio and video clocks. Expected results should verify frame ordering, segment boundaries, event positions, duration calculations, and exported metadata.
Building dependable metadata for reuse
The value of time-aware processing appears after analysis finishes. Face, logo, speaker, scene, and duplicate-content results are useful only when a user can locate the corresponding media accurately. Each result should therefore carry enough timing information to survive transfers between systems.
A strong metadata record can include the asset identifier, segment identifier, source timecode, normalized start and end positions, frame references where available, and the time basis used for the calculation. It should also indicate whether an event crosses a discontinuity. An event that begins before a reset and ends after it may need to be represented as two ranges linked by a common event ID rather than as one misleading interval.
Deduplication systems particularly benefit from this model. A repeated programme may have different starting timecodes in two files, yet its visual or audio fingerprints can reveal that the content is identical. Segment-aware fingerprints prevent a timestamp reset from being mistaken for a content change. Conversely, a real edit can be preserved as a meaningful boundary even when the surrounding material is similar.
For users, the interface should make timing understandable. A player can display source timecode while using the normalized timeline for navigation, with a visible marker at each discontinuity. Exports can include both values so that an editor can return to the original media and an archive can maintain a consistent index. These details turn anomaly handling from an internal technical fix into a practical part of the viewing and production experience.
Recommendations for implementation
- Preserve original timecode and normalized analysis time as separate fields throughout the pipeline.
- Detect discontinuities using timestamps, frame cadence, stream metadata, and decoder evidence together.
- Create explicit segment boundaries and reset stateful trackers when continuity can no longer be justified.
- Attach segment identifiers, timing basis, and confidence information to every extracted metadata event.
- Test live and file-based workflows with resets, gaps, repeated frames, variable frame rates, and concatenated sources.
Timecode discontinuities are a normal property of real broadcast workflows, not an edge case that can be removed with a single correction formula. ReCAP’s approach to real-time content analysis depends on recognising these changes, preserving their meaning, and continuing analysis without corrupting the relationship between metadata and media.
Explore the ReCAP project to follow its research, demonstrations, consortium activities, and tools for broadcast-quality video understanding. When timing remains explicit from ingestion through export, automated analysis becomes easier to audit, search, and reuse across the media production chain.