ML-Based Indexing of Media Libraries for Insights and Search

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2022, No. 10, pp. 1–17
Abstract: Recent advances in Machine Learning (ML) have produced a new form of semantic indexing that lets users enhance searches, as well as gain new insights into their media libraries. Unlike typical search systems that use extracted metadata, semantic indexing allows users to find relevant material without the need to tag the media with selections from a predefined taxonomy. With semantic search, users can simply enter unstructured text, and the system will find the best matching media clips. The paper extends the use of the same technology to gather analytics on the data which can then be further correlated to generate various insights. — This new form of media indexing can be performed with the CLIP model from OpenAI. The model encodes images and text into embeddings that can be searched to find the closest semantic similarity, enhanced with learned cultural knowledge. This type of indexing can be made practical using a database like Elasticsearch. The system has the benefit of finding media based on keywords, synonyms, and summaries. The same system can also be used for analytics and insights, such as clustering, shot detection, and creating a 2-dimensional map to display correlations. — The paper also presents extensions to the semantic search systems. Based on a study of multiple existing models, these extensions provide new capabilities - to handle many media types, using additional languages, search for spoken phrases in audio files, finding both verbatim and semantically similar phrases, extracting semantic information that leverages multiple video frames, and searching for ambient sounds.
Publication Date: 2022-10-24
DOI: 10.5594/M001973
ISBN: Print: 978-1-61482-963-8
Link: https://doi.org/10.5594/M001973
Author(s): Rob GonsalvesEngineering Fellow, Avid Technology
Zahra MontajabiConcordia University, Avid Technology
Shailendra MathurV.P. Architecture, Avid Technology
Nizar BouguilaPhD, Concordia University
Keyword(s): Artificial Intelligence, Machine Learning, Media Search, Semantic Search, OpenAI CLIP, Multilingual CLIP, Wav2vec2, CLIP4Clip, Wav2CLIP, Microsoft CLAP
Copyright: © 2022 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

M. Lapin, M. Hein, and B. Schiele, “Analysis and optimization of loss functions for multiclass, top-k, and multilabel classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7):1533–1554, July 2018 EXTERNAL
A. Radford, et al., “Learning Transferable Visual Models From Natural Language Supervision,” arXiv:2103.00020, Feb. 2021 EXTERNAL
H. Luo, et al., “CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval,” arXiv:2104.08860, May 2021 EXTERNAL
G. Salton and C. Buckley, “Term-weighting Approaches in Automatic Text Retrieval,” Information Processing and Management, 1;24(5):513-23, Jan. 1988 EXTERNAL
M. Plakal and D. Ellis, “YAMNet: A Pretrained Deep Net that Predicts 521 Audio Events,” https://github.com/tensorflow/models/tree/master/research/audioset/yamnet EXTERNAL
N. Reimers, I. Gurevych, “Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation,” 2020 Conference on Empirical Methods in Natural Language Processing (EMLP) EXTERNAL
Y.C. Chen, et al., “Uniter: Universal image-text representation learning,” European Conference on Computer Vision, Springer, Cham, 2020 EXTERNAL
H. Dong, “A Survey in Semantic Search Technologies,” 2008 2nd IEEE International Conference on Digital Ecosystems and Technologies EXTERNAL
H.H. Wu, “Wav2CLIP: Learning Robust Audio Representations From CLIP,” 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) EXTERNAL
B. Elizalde, “CLAP: Learning Audio Concepts From Natural Language Supervision,” arXiv:2206.04769, July 2012 EXTERNAL
D. Qi, et al., “Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data,” arXiv:2001.07966, Jan 2020 EXTERNAL
D. Chen and W. Dolan, “Collecting highly parallel data for paraphrase evaluation,” Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Jun. 2011 EXTERNAL
I. Croitoru, et al., “Teachtext: Crossmodally Generalized Distillation for Text-video Retrieval,” Proc. of the IEEE/CVF International Conference on Computer Vision, 2021 EXTERNAL
K. Pearson, “On Lines and Planes of Closest Fit to Systems of Points in Space,” Philosophical Magazine, 2 (11): 559–572, Nov. 1901 EXTERNAL
J. McQueen, “Some Methods for Classification and Analysis of Multivariate Observations,” Proc. 5th Berkeley Symposium on Mathematical Statistics and Probability, 1967 EXTERNAL
A. Li, et al., “Learning Visual N-grams from Web Data,” Proc. of the IEEE International Conference on Computer Vision, pp. 4183–4192, 2017 EXTERNAL
J. Tibshirani, “Text similarity search with vector fields,” Aug. 2019, https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch EXTERNAL
E. Amrani, et al., “Noise Estimation using Density Estimation for Self-supervised Multimodal Learning,” Proc. of the AAAI Conference on Artificial Intelligence, 35 (8), 2021 EXTERNAL
Y. Yu, et al., “Video Captioning and Retrieval Models with Semantic Attention,” arXiv:1610.02947, Oct. 2016 EXTERNAL
K. Qiuqiang, “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition,” IEEE/ACM Trans. Audio, Speech and Lang. Proc., 28: 2880–2894, Jan. 2020 EXTERNAL
B.A. Plummer, “Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models,” Proc. of the IEEE International Conference on Computer Vision (ICCV), pp. 2641-2649, 2015 EXTERNAL
L. van der Maaten and G. Hinton, “Visualizing Data Using t-SNE,” Journal of Machine Learning Research, 9: 2579–2605, Nov 2008 EXTERNAL
G. Li, et al., “Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training,” Proc. of the AAAI Conference on Artificial Intelligence, 34 (7): AAAI-20, Apr. 2020 EXTERNAL
T.Y. Lin, et al. “Microsoft COCO: Common Objects in Context,” European Conference on Computer Vision, Springer, Cham, 2014 EXTERNAL
A. Baevski, et al., “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” 2020 Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) EXTERNAL
M. Bain, et al., “Frozen in time: A Joint Video and Image Encoder for End-to-end Retrieval,” Proc. of the IEEE/CVF International Conference on Computer Vision, 2021. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M001973.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Rob Gonsalves, Zahra Montajabi, Shailendra Mathur, and Nizar Bouguila; ML-Based Indexing of Media Libraries for Insights and Search, SMPTE Meetings and Conferences ( October 2022); SMPTE, 2022, ISBN: 978-1-61482-963-8. Available at https://doi.org/10.5594/M001973

Snippet:

Rob Gonsalves, Zahra Montajabi, Shailendra Mathur, and Nizar Bouguila; ML-Based Indexing of Media Libraries for Insights and Search, SMPTE Meetings and Conferences ( October 2022); SMPTE, 2022, ISBN: 978-1-61482-963-8. Available at https://doi.org/10.5594/M001973

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Rob Gonsalves, Zahra Montajabi, Shailendra Mathur, and Nizar Bouguila; <cite>ML-Based Indexing of Media Libraries for Insights and Search</cite>, SMPTE Meetings and Conferences ( October 2022); SMPTE, 2022, ISBN: 978-1-61482-963-8. Available at <a href="https://doi.org/10.5594/M001973" target="_blank" rel="noopener">https://doi.org/10.5594/M001973</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Rob Gonsalves, Zahra Montajabi, Shailendra Mathur, and Nizar Bouguila; <cite id="bib-10-5594-m001973">ML-Based Indexing of Media Libraries for Insights and Search</cite>, SMPTE Meetings and Conferences ( October 2022); SMPTE, 2022, ISBN: 978-1-61482-963-8
<span class="doi">10.5594/M001973</span>
</li>

Referenced By

Machine Learning Applied to Media Libraries for Insights, Search, and Segmentation [Active]