Detection and Correction of Lip-Sync Errors Using Audio and Video Fingerprints

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 119, No. 3, pp. 42–52
Abstract: A method for measuring and maintaining time synchronization between an A/V stream (A/V) is described. A/V fingerprints are used to create a combined A/V synchronization signature (A/V Sync Signature) at a known reference point. This signature is used at later points to measure A/V timing synchronization relative to the reference point. This method may be used, for example, to automatically detect and correct A/V synchronization (i.e., lip-sync) errors in broadcast systems and other applications. Advantages of the method described over other existing methods include that it does not require modification of the audio or video signals, it can respond to dynamically changing synchronization errors, and it is designed to be robust to modifications of the A/V signals. Although the system requires data to be conveyed to the detection point, this data does not need to be synchronized with, or directly attached to, the audio or video streams. This method uses fingerprints; it also enables other fingerprinting applications within systems, such as content identification and verification. In addition, it may be used to maintain synchronization of other metadata associated with A/V streams.
Publication Date: 2010-04-01
DOI: 10.5594/J11398
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/J11398
Author(s): Kent Terry
bio
Kent Terry is currently a senior staff engineer at Dolby Laboratories in the Sound Technology Research group. Terry joined Dolby in 1992 to work on HDTV audio applications. He was part of the team that developed Dolby Digital for the ATSC, for which Dolby received an Emmy award, and he helped build the first working prototypes of the ATSC system. He was also part of the team that developed Dolby E for distribution of multichannel audio in broadcast distribution networks, also an Emmy award-winning technology. More recently, he has worked on solutions for problems in broadcasting such as lip sync. He has been active on SMPTE and ATSC standards committees, and has chaired the SMPTE group that helped revise SDI and HD-SDI embedded audio standards. Terry has been a key contributor to several SMPTE standards, including standards for conveying coded audio and other data on AES3 interfaces.
Regunathan Radhakrishnan
bio
Regunathan Radhakrishnan joined the sound technology research team at Dolby Laboratories in June 2006 and is currently working on audio and video fingerprinting. Before joining Dolby, he worked at Mitsubishi Electric Research Laboratories (MERL) as a visiting researcher. He was one of the leading contributors to MERL's Video Summarization algorithm, based on audio classification which is now a differentiating feature of Mitsubishi Electric's DVD recorder in Japan. His current research interests include statistical machine learning, audio classification, video summarization, multimedia content identification, and security. Radhakrishnan has published several conference papers, journal papers, book chapters and a book on multimedia content analysis and security.
Copyright: © 2010 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

1. Radhakrishnan R. Bauer C. Cheng C. Terry K. , “Audio Signature Extraction Based on Projections of Spectrograms,” Proc. IEEE Interntional Conference on Multimedia and Expo, Beijing, China, 2007 . EXTERNAL
2. Radhakrishnan R. Bauer C. , “Content-based Video Signatures Based on Projections of Difference Images,” IEEE Workshop on Multimedia Signal Processing, Crete, Greece, 2007 . EXTERNAL
3. Radhakrishnan R. Terry K. Bauer C. , “Audio and Video Signatures for Synchronization,” Proc. IEEE International Conf. on Multimedia and Expo, Hannover, Germany, 2008 . EXTERNAL
4. Kozat S. S. Venkatesan R. Mihcak M. K. , “Robust Perceptual Image Hashing via Matrix Invariants,” IEEE International Conference on Image Processing, Singapore, 2004 . EXTERNAL
5. Swaminathan A. Mao Y. Wu Min , “Image Hashing Resilient To Geometric and Filtering Operations,” IEEE Workshop on Multimedia Signal Processing, Siena, Italy, 2004 . EXTERNAL
6. Burges C.J.C. Platt J. C. Jana S. , “Distortion Discriminant Analysis for Audio Fingerprinting,” IEEE Transactions on Speech and Audio Processing, May 2003 . EXTERNAL
7. Haitsma J. Kalker T. , “A Highly Robust Audio Fingerprinting System,” Proc. International Conference on Music Information Retrieval, Paris, France, 2002 . EXTERNAL
8. Lu Chun-Shien , “Audio Fingerprinting Based on Analyzing Time-frequency Localization of Signals,” IEEE Workshop on Multimedia Signal Processing, St. Thomas, Virgin Islands, 2002 . EXTERNAL
9. Batlle E. Masip J. Guaus E. , “Automatic Song Identification in Noisy Broadcast Audio,” Proc. Signal and Image Processing , Aug. 2002 . EXTERNAL
10. Fridrich J. Goljan M. , “Robust Hash Functions for Digital Watermarking,” Proc. of International Symposium on Information Technology, Las Vegas, 2000 . EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-J11398.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Kent Terry and Regunathan Radhakrishnan; Detection and Correction of Lip-Sync Errors Using Audio and Video Fingerprints, SMPTE Motion Imaging Journal ( Volume: 119, Issue: 3, April 2010); SMPTE, 2010. Available at https://doi.org/10.5594/J11398

Snippet:

Kent Terry and Regunathan Radhakrishnan; Detection and Correction of Lip-Sync Errors Using Audio and Video Fingerprints, SMPTE Motion Imaging Journal ( Volume: 119, Issue: 3, April 2010); SMPTE, 2010. Available at https://doi.org/10.5594/J11398

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Kent Terry and Regunathan Radhakrishnan; <cite>Detection and Correction of Lip-Sync Errors Using Audio and Video Fingerprints</cite>, SMPTE Motion Imaging Journal ( Volume: 119, Issue: 3, April 2010); SMPTE, 2010. Available at <a href="https://doi.org/10.5594/J11398" target="_blank" rel="noopener">https://doi.org/10.5594/J11398</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Kent Terry and Regunathan Radhakrishnan; <cite id="bib-10-5594-j11398">Detection and Correction of Lip-Sync Errors Using Audio and Video Fingerprints</cite>, SMPTE Motion Imaging Journal ( Volume: 119, Issue: 3, April 2010); SMPTE, 2010
<span class="doi">10.5594/J11398</span>
</li>