Machine Learning Applied to Media Libraries for Insights, Search, and Segmentation

Bibliographic Reference(s)

• ML-Based Indexing of Media Libraries for Insights and Search [Active]
• L. van der Maaten and G. Hinton, “Visualizing Data Using t-SNE,” J. Mach. Learn. Res., 9:2579–2605, 2008. Conclusion Semantic indexing is a new method of searching for and retrieving media files that uses ML to extract and encode semantic representations found in various media formats. This allows for searches using unstructured text queries, which can be faster and more accurate than traditional metadata searches. Semantic representations can also be used to provide insights into the media library. The CLIP model from OpenAI is one example of a model that can be used for semantic indexing, and the Elasticsearch database can be used to store the resulting embeddings. There are many potential applications for semantic search, including finding media with queries that contain summaries or synonyms, using learned cultural knowledge. Additionally, semantic search can be extended to use multiple languages for queries and used to find additional media types like video clips and audio files. Next Steps Recently, the EBU, Movie labs, and SMPTE published a primer29 on using semantic technologies, which, among the various uses, lists contextual search and discovery as one of the benefits. The approach outlined in that primer discusses using a knowledge graph of triples stored in graph databases to relate semantically linked data. This approach is an alternate, if not complementary, approach to semantic search using embeddings. Prior to the primer mentioned earlier, Wei et al.30 provided a comprehensive overview of semantic search systems based on linked data. Semantically linked systems provide for usage that can go beyond conventional search usage. The entity-relationship graph lends itself to queries that go beyond resolving a search query to a digital asset. The entity relationships can be used to discover hitherto unknown relationships between data records and digital assets by traversing edges in the graph to provide insights and recommendations;31 to organize the data in predefined structures, making traversal and automation possible; to connect heterogeneous and distributed data sources while maintaining data integrity. Most of these knowledge graph-based systems require standardized predefined ontologies and a-priori-defined taxonomies to model the relationships. In contrast, text embedding-based systems can discover the taxonomy. Next steps include comparing the architectures and results of using semantic embeddings versus this new approach to knowledge graphs for searching media libraries. Part of the investigation is to evaluate if the technologies complement each other rather than being distinct approaches. EXTERNAL
• M. Plakal and D. Ellis, “YAMNet: A Pretrained Deep Net that Predicts 521 Audio Events,” 2022. EXTERNAL
• H. Luo et al., “CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval,” 2021, arXiv:2104.08860. EXTERNAL
• D. Chen and W. Dolan, “Collecting Highly Parallel Data for Paraphrase Evaluation,” Proc. of the 49th Annu. Meeting of the Assoc. for Computational Linguistics: Human Language Technol., pp. 190–200, 2011. EXTERNAL
• K. Qiuqiang, “PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition,” IEEE/ACM Trans. Audio, Speech, Language Process., 28:2880–2894, 2020. EXTERNAL
• M. Lapin, M. Hein, and B. Schiele, “Analysis and Optimization of Loss Functions for Multiclass, Top-k, and Multilabel Classification,” IEEE Trans. Pattern Analysis Mach. Intell., 40(7):1533–1554, 2018. EXTERNAL
• J. Chicaiza and P. Valdiviezo-Diaz, “A Comprehensive Survey of Knowledge Graph-Based Recommender Systems: Technologies, Development, and Contributions,” Information, 12 (6):232, May 2021. EXTERNAL
• T. Y. Lin et al., “Microsoft COCO: Common Objects in Context,” in European Conference on Computer Vision, Springer: Cham, Switzerland, 2014. EXTERNAL
• J. McQueen , “Some Methods for Classification and Analysis of Multivariate Observations,” Proc. 5th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281–297, 1967. EXTERNAL
• D. Qi et al., “Cross-Modal Pre-Training With Large-Scale Weak-Supervised Image-Text Data,” 2020, arXiv:2001.07966. EXTERNAL
• Y. C. Chen et al., “Uniter: Universal Image-Text Representation Learning,” in European Conference on Computer Vision, Springer: Cham, Switzerland, 2020. EXTERNAL
• Y. Yu et al., “Video Captioning and Retrieval Models with Semantic Attention,” 2016, arXiv:1610.02947. EXTERNAL
• B. Elizalde, “CLAP: Learning Audio Concepts From Natural Language Supervision,” 2012, arXiv:2206.04769. EXTERNAL
• A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” 2021, arXiv:2103.00020. EXTERNAL
• G. Salton and C. Buckley, “Term-Weighting Approaches in Automatic Text Retrieval,” Inform. Process. Manag., 24(5):513– 523, Jan. 1988. EXTERNAL
• W. Wei et al., “Search With Meanings: An Overview of Semantic Search Systems,” Int. J. Commun. SIWN, 3:76–82, 2008. EXTERNAL
• M. Bain et al., “Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval,” Proc. of the IEEE/CVF Int. Conf. on Comput. Vision, Apr. 2021. EXTERNAL
• G. Li et al., “Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training,” Proc. of the AAAI Conf. on Artificial Intell. (AAAI-20), vol. 34, no. 7, 2020. EXTERNAL
• N. Reimers and I. Gurevych, “Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation,” 2020 Conf. on Empirical Methods in Natural Language Process. (EMLP), 2020. EXTERNAL
• A. Baevski et al., “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” 2020 Proc. Conf. on Neural Inform. Process. Syst. (NeurIPS), 2020. EXTERNAL
• European Broadcasting Union (EBU), Movie Labs, SMPTE, Media in the Cloud: Ontology and Semantic Web Technology Navigation Guide, 2022. EXTERNAL
• I. Croitoru et al., “Teachtext: Crossmodally Generalized Distillation for Text-Video Retrieval,” Proc. of the IEEE/CVF Int. Conf. on Comput. Vision, 2021. EXTERNAL
• X. Zou, “A Survey on Application of Knowledge Graph,” J. Phys. Conf. Ser., 1487(1):012016, 2020. EXTERNAL
• A. Li et al., “Learning Visual N-Grams From Web Data,” Proc. of the IEEE Int. Conf. on Comput. Vision, pp. 4183–4192, 2017. EXTERNAL
• H. H. Wu, “Wav2CLIP: Learning Robust Audio Representations From CLIP,” 2022 IEEE Int. Conf. Acoustics, Speech and Signal Process. EXTERNAL
• J. Tibshirani, “Text Similarity Search With Vector Fields,” Aug. 2019. Accessed: Feb. 2023. [Online]. Available: https://www.elastic.co EXTERNAL
• E. Amrani et al., “Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning,” Proc. of the AAAI Conf. on Artificial Intell., 35(8):6644–6652, 2021. EXTERNAL
• K. Pearson, “On Lines and Planes of Closest Fit to Systems of Points in Space,” Philos. Mag., 2(11):559–572, 1901. EXTERNAL
• B. A. Plummer, “Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models,” Proc. of the IEEE Int. Conf. on Comput. Vision (ICCV), pp. 2641– 2649, 2015. EXTERNAL
• H. Dong, “A Survey in Semantic Search Technologies,” 2008 2nd IEEE Int. Conf. on Digital Ecosystems and Technol., 2008. EXTERNAL