Project Brief (Draft)

Aim

Realtime Content Analysis & Processing (ReCAP) is a Machine Learning and Computer Vision project, co-funded by the European Union’s Horizon 2020 programme. The project aims to produce a commercial software platform that can automatically analyze and extract time-stamped, descriptive and technical metadata, in real-time, from live broadcast-quality video content and to process existing archive content.

Product Description

ReCAP utilizes processing algorithms from the consortium partners which have been optimised for real time GPU processing, with optional broadcast quality SDI and IP recording, supporting growing file ingest and the ability to batch process existing media content (e.g. archive.)

“Customers with fast turn around live video workflows or substantial video archives don’t have enough time to watch, analyze, and make decisions about their content.”

“ReCAP will provide a range of automatic content-analysis services to form an affordable, scalable, open, and flexible platform designed to enhance metadata-driven media and compliance related  workflows and realize the value of content archives.”

Consortium

  • British media technology solutions provider New Media Research (NMR), which specialises in Media Asset Management (MAM) integration and custom software development.

  • ToolsOnAir, an Austrian developer of applications that integrate broadcast quality ingest, scheduling and playout automation, with real-time graphics, workflow automation and shared storage.

  • Nablet, a German developer of streaming, codec, muxing, analysis and transcoding solutions.

  • Austrian non-profit Joanneum Research Institute (JRS), focuses on applied research and development around Artificial Intelligence and Machine Learning technologies.

Funding

The ReCAP project is 75% co-funded by a €1 million grant from Horizon 2020, the European Union research and innovation fund that supports making technology commercially available and affordable for small to medium-size enterprises.

 

Functionality

The ReCAP product will deliver time-stamped, real-time descriptive analysis of video and audio content and may include the following functionality, which exists within the consortium;

  • Automatic Speech Recognition (ASR)

  • On-screen text extraction (OCR)

  • Logo detection

  • Face detection

  • Face identification

  • Content fingerprinting

  • Content duplication

  • Content similarity

  • Language detection*

  • Instance search*

  • Image classification*

  • Music identification**

Technical Quality Analysis functionality includes;

  • Automatic detection of quality impairments, such as lost/frozen video frames

  • Block dropouts

  • Scene change detection

  • Visible macroblocking

  • Estimation of sharpness and noise

  • Black detection

  • Colour bar detection

  • Tone detection

  • Silence detection

  • Potentially automatic quality improvement

  • Interpolation of image regions or entire frames

  • Denoising

  • Block artifact suppression

* Future release

** Future third-party integration

Use Cases

Archive Content Enrichment

Automatically adding descriptive and technical metadata to legacy archive content, provide time stamped metadata search to repurpose existing content, avoid the need to reshoot content and provide opportunities to license and commercialise historical content for news, documentaries, re-broadcast and VOD.

Compliance

Improve compliance related video workflows, typically;

  • Broadcast - identify and remove inappropriate content prior to transmission

  • Financial - identify who said what/when for investigative purposes

  • Legal - qualify, preserve or expunge sensitive content from the legal record

To assist production teams and compliance officers, to reduce time and effort to find and correct/remove inappropriate content.

Rights Management

Significant media management challenges exist for users of 3rd party content, specifically where content contains rights managed materials. Whilst ReCAP does not aim to provide rights management functionality directly, the consortium is mindful that automated metadata enrichment can identify and fingerprint content to help reduce the risk of using content with expired rights and thus reduce the risk of rights violations and potentially damaging legal action.

Architecture

ReCAP algorithms are optimised to use the GStreamer open source pipeline-based multimedia framework, which allows processes to be GPU accelerated, providing optional recording and processing of broadcast quality live video feeds in real time (SDI, IP stream or growing file ingest).

Batch processing of existing media files will also facilitate analysis of existing archive content.

Harvested time stamped metadata will be stored as JSON, using a FIMS/EBU compliant metadata structure, supporting the MPEG7 standard.

ReCAP aims to build a modular and scalable software platform, that can be virtualised and will be cloud agnostic, which is flexible (e.g. proxy workflows) and provides potential integration with other FOSS or commercial 3rd party Machine Learning and Computer Vision technologies, via RESTful APIs.

NAB 2017

Exhibiting in the Future Park at NAB 2017, a prototype of ReCAP was announced and demonstrated.

Diagram of software architecture demonstrated at NAB:

ReCapSysArch_NAB.jpg

 

Diagram of full software architecture;ReCapSysArch_FULL.jpg

 

Software Deployment

ReCAP runs on COTS IT hardware and Linux OS, it can be virtualised and packaged using configuration management for easy deployment.

The ReCAP product will support on-premise, data center, and cloud deployment models - hybrid cloud and auto scale i.e. can go from single server on-premise server or multiple VMs to unlimited cloud scale processing with spin up/down services on-demand.

Software Integration

Be able to process broadcast-quality video up to 4K/UHD (MXF or QuickTime recording), with output immediately available for review in an HTML5 browser based media player and multi-track metadata timeline, integrated with commercial Media Asset Management (MAM) software (Cantemo Portal and Vidispine APIs for first release) and later workflow orchestration tools and transcoding software, integrated by means of open and documented RESTful APIs.

 

 

ReCAP Web Player Working Prototype

 

Licensing Model*

The product will likely be available in 3 options, depending upon customers needs;

  • Phase 1 - on-premise, high performance turnkey appliance with perpetual software licensing and modular options - fast, faster, fastest hardware models (optional SDI/IP ingest recording ports), for clients wanting to process their valuable or sensitive media content in-house.

  • Phase 2 - perpetual software license Virtual Machine (VM) - for clients with existing CPU & GPU infrastructure on-premise or in their data centres.

  • Phase 3 - available as Platform-as-a-Service (PaaS) with Bring-Your-Own-Cloud (BYOC) - for clients with existing media on cloud storage (e.g. AWS, Azure) or wanting to burst cloud compute.

  • Annual software maintenance and 2nd/3rd line SLA will be provided by local channel partners and the vendor.

  • Custom software development and 3rd party integrations available on request as Professional Services to the vendor (time & materials standard daily rate).

  • API licensing may also be made available to 3rd party vendors for OEM and integration into their product lines.

*Subject to change

 

Commercialisation

As the primary vendor, NMR will be responsible for managing commercialisation, marketing and future software development, while ToolsOnAir, Nablet and JRS will provide their technology and future functionality via an OEM license agreement.

The finished product from the ReCAP Project will be available exclusively through NMR for an initial period, for direct engagement with prospective users, to provide comprehensive professional services and to deliver successful project outcomes.

NMR will then partner and train a select group of global channel partners, including Value Added Resellers (VAR) and Systems Integrators (SI) to facilitate regional sales and provide local 1st/2nd line technical support and escalation to the vendor.

Industry Stakeholder Group

A stakeholder group of over 20 broadcasters, publishers, brands, corporate enterprises and legal institutions, are working closely with the project implementation team, to state their requirements, provide test footage and feedback, check project progress and run pilot/alpha/beta tests, ensuring the final product meets the industry’s need for automatic media content analysis and that it satisfies real world use cases and workflow challenges.

Project Timeframe

Commenced December 2016

POC NAB 2017

Alpha IBC 2017

Beta NAB 2018

First Release June 2018

Public Presentations

ReCAP Project public presentations will be at IBC 2017 (Vidispine booth 3.A23), NAB New York (PVT booth TBC), BVE 2018 (NMR booth TBC), SXSWi 2018 and NAB 2018 (booth TBC).

Keep In Touch

The ReCAP project website (www.recap-project.com) will be regularly updated with blog postings, white papers, and online video demonstrations. Updates will also be posted on Twitter (@ReCAP_Project). For more information, please contact Neil Anderson +44 (0)20 7255 2700 or email:  info [at] recap-project.com