Processing live streams

In certain scenarios it's critical to index and search multimedia content it realtime. An example could be live video streams being broadcast by end users to a wide audience. In certain cases such live streams might contain content that violates certain terms of service. It's not uncommon to run into live video streams with copyrighted movies or live sport events. Another potential problem could be objectionable content, such as nudity, porn terrorism, etc. It's extremely important to identify in a timely manner and undertake all the necessary actions to minimize the potential damage.

Vertx can index and search live stream data in real-time manner. Please, find a demo in the following video.

To register live content, one has to have access to a private bucket. Given that a live stream we are going to index is broadcast at the ${BROADCAST_URL}, the following command will start processing live stream and registering chunks of fingerprints in real time:

vertx register \
    --streaming \
    --bucket=${BUCKET} \
    --meta_title='streaming_title' \
    --meta_album 'streaming_album'  \
    --meta_artist 'streaming_artist' \
    --meta_cover_url '/images/streaming.jpg' \
    --meta_imdb_id 43 \
    --meta_year 2010 \
    --meta_extra 'streaming_meta' \
  ${BROADCAST_URL}

Every two seconds client reports successful or failed attempt to insert a chunk of data to Vertx indices:

{"bucket":"rough-bush","duration":2.0,"error":null,"media_type":"video","offset":0.0,"request_id":"5ea7a07d-4095-4098-9d32-110c03e961b1","source_uid":"5554329625350814294","status":"OK"}
{"bucket":"rough-bush","duration":2.0,"error":null,"media_type":"audio","offset":0.0,"request_id":"5ea7a07d-4095-4098-9d32-110c03e961b1","source_uid":"5554329625350814294","status":"OK"}
...

Signatures for all the broadcast content is stored in the Vertx indices for 48 hours. If a longer TTL is required, the content has to be registered as a static multimedia asset after the broadcast is ended.

Vertx CLI can also be used to perform content-based search providing ${BROADCAST_URL} as the input:

vertx search \
  --streaming \
  --search_buckets=${BUCKET} \
  ${BROADCAST_URL}

When processing data in streaming mode, vertx client prints intermediate results to the standard output pipe every five seconds:

{"duration":5.0625,"error":null,"matches":[{"metadata":{"album":"streaming_album","artist":"streaming_artist","bucket":"rough-bush","cover_url":"/images/streaming.jpg","duration":24.9375,"extra":"streaming_meta","imdb_id":43,"title":"streaming_title","uid":"5554329625350814294","year":2010},"segments":[{"duration":5.0,"que_offset":0.0,"ref_offset":0.0}]},{"metadata":{"album":"sample_album","artist":"sample_artist","bucket":"rough-bush","cover_url":"/images/cover.jpg","duration":24.9375,"extra":"extra_meta","imdb_id":4423,"title":"sample_title","uid":"1758730481226206085","year":2004},"segments":[{"duration":5.0,"que_offset":0.0,"ref_offset":0.0}]}],"media_type":"video","request_id":"d21c72f2-22eb-4604-b3cd-26356a5761ca","source_path":"udp://localhost:5000/1758730481226206085","source_uid":"5554329625350814294","status":"OK"}
{"duration":5.875,"error":null,"matches":[{"metadata":{"album":"sample_album","artist":"sample_artist","bucket":"rough-bush","cover_url":"/images/cover.jpg","duration":24.9375,"extra":"extra_meta","imdb_id":4423,"title":"sample_title","uid":"1758730481226206085","year":2004},"segments":[{"duration":5.8125,"que_offset":0.0,"ref_offset":0.0}]},{"metadata":{"album":"streaming_album","artist":"streaming_artist","bucket":"rough-bush","cover_url":"/images/streaming.jpg","duration":24.9375,"extra":"streaming_meta","imdb_id":43,"title":"streaming_title","uid":"5554329625350814294","year":2010},"segments":[{"duration":5.8125,"que_offset":0.0,"ref_offset":0.0}]}],"media_type":"audio","request_id":"d21c72f2-22eb-4604-b3cd-26356a5761ca","source_path":"udp://localhost:5000/1758730481226206085","source_uid":"5554329625350814294","status":"OK"}
...

Once the stream is finished, aggregate results will be printed to the standard output for both, audio and video matches.