How ReCAP identifies and logs on-screen QR codes in video
Broadcast video contains far more information than dialogue, scenes, and moving images. A QR code may appear briefly in a television advertisement, during a live event, in a news report, or inside a piece of user-generated footage. Although viewers can scan it with a phone, media systems need a different method to find, describe, and preserve that information at scale.
ReCAP addresses this problem through real-time content analysis and processing. Its video intelligence tools examine visual material as it arrives, identify meaningful elements, and create metadata that can be searched or connected to production workflows. In this setting, detecting a QR code is not an isolated computer-vision trick. It is part of a wider process that links an on-screen object to its position, duration, confidence, and surrounding broadcast context.
The resulting record can help broadcasters understand when a QR code appeared, where it was placed, and whether it remained readable throughout a scene. It can also support media asset management, compliance checks, advertising analysis, and later retrieval of content from large archives.
Finding candidate regions in each video frame
The process begins with a stream of decoded video frames. ReCAP can sample frames at a suitable interval or inspect them continuously, depending on the required precision, processing budget, and type of content. A live broadcast may require low-latency analysis, while an archive can be processed more thoroughly after ingestion.
A QR detector first searches for visual patterns associated with two-dimensional codes. These patterns include strong contrast, square geometry, finder markers, and the regular arrangement of dark and light modules. The system does not need to treat every frame as an entirely unrelated image. It can use information from neighbouring frames to determine whether a candidate is persistent, newly introduced, moving with the camera, or part of an animated graphic.
Pre-processing improves the odds of finding codes under real broadcast conditions. Resizing, contrast adjustment, sharpening, and perspective correction can make a small or oblique code easier to interpret. The analysis may also account for letterboxing, overlays, compression artefacts, changing brightness, and motion blur. These steps are important because a code that is clear to a viewer for several seconds may be difficult to decode in any single frame.
Separating a real QR code from visual noise
Detection and decoding are related but distinct tasks. Detection asks whether a QR-like object exists in an image and where it is located. Decoding attempts to recover the payload, which may be a web address, campaign identifier, text string, or another structured value. A detector can therefore report a visible code even when the image quality is insufficient to read its contents.
This distinction allows ReCAP to retain useful metadata instead of treating an unsuccessful decode as a complete failure. A record might state that a QR-shaped object was present, include its bounding box and confidence score, and mark the payload as unavailable. If later frames provide a sharper view, the system can use them to improve the event record.
False positives are controlled by combining several signals rather than relying on a single square pattern. The system can compare the candidate’s geometry with the expected structure of a QR code, check whether it persists across frames, and assess whether a decoder consistently recognizes the same payload. Logos, checkerboard graphics, subtitles, and decorative squares may resemble parts of a code but generally fail this combined examination.
Context also matters. A code embedded in an advertisement may appear in a stable corner, while a code printed on a moving object may change size and orientation. Temporal tracking helps maintain the identity of a candidate as it moves, disappears briefly, or is temporarily obscured by an edit or graphic transition.
Recovering payloads and creating reliable event records
When a QR code can be read, its payload becomes a valuable part of the metadata. A URL can be stored as text, while a non-web payload can be preserved in its original form. The system can also retain a normalized representation for search and analytics, provided that normalization does not alter the original value needed for verification.
A useful event record goes beyond the decoded string. It can include the video asset or stream identifier, timecode of first appearance, timecode of last observed appearance, frame references, image coordinates, detection confidence, decoding status, and the number of frames that supported the result. These fields make the result actionable for editors, archivists, and monitoring teams.
The Joanneum Research team contributes to the wider research environment in which this kind of computer-vision analysis is developed and evaluated. Linking QR recognition to broader metadata extraction helps ensure that the output can fit into practical broadcast and media-asset workflows rather than remaining a standalone laboratory result.
A stable identifier can connect repeated observations of the same code into one appearance event. For example, a code visible across 180 frames should not create 180 unrelated database entries. Instead, those observations can be grouped, while the system retains frame-level evidence when detailed review is necessary.
Tracking a code across time and scenes
Video changes continuously, so an effective detector needs temporal logic. Once a QR code has been located, tracking can follow its position from frame to frame. This reduces duplicate detections and helps estimate the exact period during which the code was visible.
A tracking record may include a start time, end time, duration, average position, and trajectory. It can also show whether the object stayed fixed as part of a channel graphic or moved with a physical surface. These distinctions support later analysis: a persistent corner overlay has different editorial significance from a code briefly shown on a product package.
Scene boundaries require special care. A code can vanish because a shot changed, because a transition covered it, or because its contrast temporarily collapsed. The system can use shot-change information, visibility gaps, and confidence thresholds to decide whether a new observation belongs to an existing event or represents a new appearance.
The process can also measure readability over time. A code may be visible for ten seconds but decodable during only three of them. Recording both visibility and successful decoding gives media teams a more accurate picture of the viewer experience. It can reveal whether a call to action stayed on screen long enough and at sufficient quality to be scanned.
| Metadata field | What it records | Why it matters |
|---|---|---|
| Asset or stream ID | The source video connected to the detection | Links the event to an archive or live channel |
| First and last timecode | When the code appeared | Supports timeline search and campaign verification |
| Payload | Decoded URL or text value | Enables content lookup and reporting |
| Bounding box | Position and size within the frame | Shows placement and helps review the source image |
| Detection confidence | Strength of the visual recognition result | Helps separate reliable findings from uncertain ones |
| Decode status | Whether the payload was successfully recovered | Distinguishes visibility from readability |
| Supporting frames | Images or references used as evidence | Allows human validation and audit trails |
| Tracking information | Movement, persistence, and interruptions | Groups observations into meaningful events |
Handling difficult broadcast conditions
On-screen QR codes are often small relative to the full frame. A high-definition source does not guarantee a high-resolution code: if the graphic occupies only a few dozen pixels, there may be too little detail to reconstruct its modules. Downstream encoding, scaling, and platform compression can make the problem harder.
Broadcast graphics can also introduce transparency, animation, glow effects, and rapidly changing backgrounds. Perspective distortion is common when a code is printed on a sign, product, or screen shown at an angle. Reflections and glare may remove parts of the pattern, while motion blur can affect the finder markers that a decoder relies on.
ReCAP’s analysis can address these conditions through multiple observations and quality-aware processing. Candidate frames may be ranked by sharpness, size, contrast, and geometric stability before decoding is attempted. Perspective normalization can transform an angled code into a more regular view, while image enhancement can recover useful detail without pretending that missing information is present.
Quality assessment is especially important for broadcast operations. A system should expose uncertainty rather than report every result as equally dependable. Confidence scores, evidence images, and decode status allow operators to establish review policies. High-confidence detections can move directly into metadata systems, while ambiguous events can be flagged for manual inspection.
The same principle supports monitoring across distribution channels. If a code is readable in the master file but not after transmission or platform delivery, comparative analysis can identify a quality problem. QR recognition therefore becomes an indirect indicator of graphics legibility and encoding performance.
Connecting QR metadata to media workflows
Once generated, QR-code metadata can be used by several teams. Editors may search an archive for every asset containing a particular campaign link. Producers can verify that a promotional code appeared during the intended segment. Compliance teams can review whether a required disclosure or interactive prompt was present and visible.
In a media asset management system, the event can become a searchable annotation attached to an asset timeline. Queries might filter content by decoded payload, date range, programme, channel, location, or duration. Thumbnail images and timecodes make it possible to open the exact moment instead of reviewing an entire programme manually.
For live production, low-latency notifications may be more useful than a detailed archive record. An operator could receive an alert when an unexpected code appears, when a planned code fails to remain visible, or when a code is detected in a region where it should not occur. The appropriate response depends on the workflow and the confidence threshold selected.
Privacy and governance should also be considered. A QR code may lead to a tracking URL or contain information associated with a campaign or individual. Systems should define how payloads are stored, who can access them, and whether sensitive values need masking or retention limits. Clear provenance is equally important: users should know whether a value was decoded automatically, verified by an operator, or inferred from repeated observations.
Measuring performance and improving the pipeline
Evaluation should measure more than the number of detected codes. Precision indicates how many reported detections are genuine, while recall indicates how many real codes were found. Decode success adds another dimension by showing how often the system recovered the actual payload rather than merely locating a QR-like region.
Testing should cover representative broadcast material: studio programmes, sports coverage, advertisements, news footage, archival recordings, animated graphics, and low-bitrate streams. Different code sizes, colours, angles, occlusion levels, and display durations should be included. A detector that performs well on clean promotional footage may behave differently on a compressed live feed.
Human review remains useful for building trustworthy evaluation sets. Annotators can mark the presence, position, visibility interval, and readability of each code. These references allow the processing pipeline to be tuned against real editorial expectations instead of an abstract image benchmark.
Operational feedback can then guide improvement. Repeated false positives may reveal a particular channel graphic that needs special handling. Missed detections may point to a sampling interval that is too wide or a minimum-size threshold that is too strict. Over time, event logs provide evidence for adjusting processing speed, confidence rules, and storage requirements.
Recommended practices for dependable QR-code metadata
- Preserve both the original payload and a normalized copy so searches remain convenient without losing forensic accuracy.
- Store timecodes, frame evidence, and bounding boxes alongside the decoded value for transparent verification.
- Distinguish visible, detected, and successfully decoded states in the metadata model.
- Use temporal tracking to combine repeated frame detections into a single appearance event.
- Route low-confidence or incomplete results to review instead of discarding them silently.
For ReCAP, QR recognition illustrates how real-time video analysis can turn fleeting visual details into structured information. The value lies in the complete chain: locating the code, testing its readability, tracking its appearance, recording uncertainty, and connecting the result to a searchable media workflow.
Explore ReCAP’s research and demonstrations to see how automated metadata extraction can support broadcast monitoring, production decisions, and richer media archives. Bring QR-code event logging into your video pipeline to make interactive graphics easier to verify, find, and manage.