Project Brief (Draft)
Realtime Content Analysis & Processing (ReCAP) is a Machine Learning and Computer Vision project, co-funded by the European Union’s Horizon 2020 programme. The project aims to produce a commercial software platform that can automatically analyze and extract time-stamped, descriptive and technical metadata, in real-time, from live broadcast-quality video content and to process existing archive content.
ReCAP utilizes processing algorithms from the consortium partners which have been optimised for real time GPU processing, with optional broadcast quality SDI and IP recording, supporting growing file ingest and the ability to batch process existing media content (e.g. archive.)
“Customers with fast turn around live video workflows or substantial video archives don’t have enough time to watch, analyze, and make decisions about their content.”
“ReCAP will provide a range of automatic content-analysis services to form an affordable, scalable, open, and flexible platform designed to enhance metadata-driven media and compliance related workflows and realize the value of content archives.”
British media technology solutions provider New Media Research (NMR), which specialises in Media Asset Management (MAM) integration and custom software development.
ToolsOnAir, an Austrian developer of applications that integrate broadcast quality ingest, scheduling and playout automation, with real-time graphics, workflow automation and shared storage.
Nablet, a German developer of streaming, codec, muxing, analysis and transcoding solutions.
Austrian non-profit Joanneum Research Institute (JRS), focuses on applied research and development around Artificial Intelligence and Machine Learning technologies.
The ReCAP project is 75% co-funded by a €1 million grant from Horizon 2020, the European Union research and innovation fund that supports making technology commercially available and affordable for small to medium-size enterprises.
The ReCAP product will deliver time-stamped, real-time descriptive analysis of video and audio content and may include the following functionality, which exists within the consortium;
Automatic Speech Recognition (ASR)
On-screen text extraction (OCR)
Technical Quality Analysis functionality includes;
Automatic detection of quality impairments, such as lost/frozen video frames
Scene change detection
Estimation of sharpness and noise
Colour bar detection
Potentially automatic quality improvement
Interpolation of image regions or entire frames
Block artifact suppression
* Future release
** Future third-party integration
Archive Content Enrichment
Automatically adding descriptive and technical metadata to legacy archive content, provide time stamped metadata search to repurpose existing content, avoid the need to reshoot content and provide opportunities to license and commercialise historical content for news, documentaries, re-broadcast and VOD.
Improve compliance related video workflows, typically;
Broadcast - identify and remove inappropriate content prior to transmission
Financial - identify who said what/when for investigative purposes
Legal - qualify, preserve or expunge sensitive content from the legal record
To assist production teams and compliance officers, to reduce time and effort to find and correct/remove inappropriate content.
Significant media management challenges exist for users of 3rd party content, specifically where content contains rights managed materials. Whilst ReCAP does not aim to provide rights management functionality directly, the consortium is mindful that automated metadata enrichment can identify and fingerprint content to help reduce the risk of using content with expired rights and thus reduce the risk of rights violations and potentially damaging legal action.
ReCAP algorithms are optimised to use the GStreamer open source pipeline-based multimedia framework, which allows processes to be GPU accelerated, providing optional recording and processing of broadcast quality live video feeds in real time (SDI, IP stream or growing file ingest).
Batch processing of existing media files will also facilitate analysis of existing archive content.
ReCAP aims to build a modular and scalable software platform, that can be virtualised and will be cloud agnostic, which is flexible (e.g. proxy workflows) and provides potential integration with other FOSS or commercial 3rd party Machine Learning and Computer Vision technologies, via RESTful APIs.
Diagram of software architecture demonstrated at NAB:
Diagram of full software architecture;
ReCAP runs on COTS IT hardware and Linux OS, it can be virtualised and packaged using configuration management for easy deployment.
The ReCAP product will support on-premise, data center, and cloud deployment models - hybrid cloud and auto scale i.e. can go from single server on-premise server or multiple VMs to unlimited cloud scale processing with spin up/down services on-demand.
Be able to process broadcast-quality video up to 4K/UHD (MXF or QuickTime recording), with output immediately available for review in an HTML5 browser based media player and multi-track metadata timeline, integrated with commercial Media Asset Management (MAM) software (Cantemo Portal and Vidispine APIs for first release) and later workflow orchestration tools and transcoding software, integrated by means of open and documented RESTful APIs.
ReCAP Web Player Working Prototype
The product will likely be available in 3 options, depending upon customers needs;
Phase 1 - on-premise, high performance turnkey appliance with perpetual software licensing and modular options - fast, faster, fastest hardware models (optional SDI/IP ingest recording ports), for clients wanting to process their valuable or sensitive media content in-house.
Phase 2 - perpetual software license Virtual Machine (VM) - for clients with existing CPU & GPU infrastructure on-premise or in their data centres.
Phase 3 - available as Platform-as-a-Service (PaaS) with Bring-Your-Own-Cloud (BYOC) - for clients with existing media on cloud storage (e.g. AWS, Azure) or wanting to burst cloud compute.
Annual software maintenance and 2nd/3rd line SLA will be provided by local channel partners and the vendor.
Custom software development and 3rd party integrations available on request as Professional Services to the vendor (time & materials standard daily rate).
API licensing may also be made available to 3rd party vendors for OEM and integration into their product lines.
*Subject to change
As the primary vendor, NMR will be responsible for managing commercialisation, marketing and future software development, while ToolsOnAir, Nablet and JRS will provide their technology and future functionality via an OEM license agreement.
The finished product from the ReCAP Project will be available exclusively through NMR for an initial period, for direct engagement with prospective users, to provide comprehensive professional services and to deliver successful project outcomes.
NMR will then partner and train a select group of global channel partners, including Value Added Resellers (VAR) and Systems Integrators (SI) to facilitate regional sales and provide local 1st/2nd line technical support and escalation to the vendor.
Industry Stakeholder Group
A stakeholder group of over 20 broadcasters, publishers, brands, corporate enterprises and legal institutions, are working closely with the project implementation team, to state their requirements, provide test footage and feedback, check project progress and run pilot/alpha/beta tests, ensuring the final product meets the industry’s need for automatic media content analysis and that it satisfies real world use cases and workflow challenges.
Commenced December 2016
POC NAB 2017
Alpha IBC 2017
Beta NAB 2018
First Release June 2018
Keep In Touch
The ReCAP project website (www.recap-project.com) will be regularly updated with blog postings, white papers, and online video demonstrations. Updates will also be posted on Twitter (@ReCAP_Project). For more information, please contact Neil Anderson +44 (0)20 7255 2700 or email: info [at] recap-project.com