Customized Facial Expression Analysis in Video

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 131, No. 3, pp. 17–24
Abstract: Existing computer vision-based emotion recognition systems are trained to classify images of faces into a very limited number of emotions. In this work, we take a different approach and train a convolutional neural network (CNN) that is able to distinguish subtle differences in facial expressions appearing in both individual images and videos. For this effect, we train a feature embedding network that maps a facial image into a position in embedding space, such that it is positioned close to similar facial expressions compared with other expressions. We train the feature embedding network using triplet loss on the publicly available facial expression comparison (FEC) data set. The proposed facial expression model is lightweight (4.7M parameters) and obtains a triplet prediction accuracy of 84.5%—very close to the average human performance of 86.2%. A key contribution of this work is adapting the model that has been trained on images to work reliably on video content. To this end, we propose a novel automatic face quality assessment (FaceQA), which allows us to filter out faces that are unsuited for expression analysis. We validate the proposed approach on several applications, including searching video content for a specific expression from a single query face, generating facial expression statistics of various actors appearing in a video, as well as generating facial expression summaries.
Publication Date: 2022-04-01
DOI: 10.5594/JMI.2022.3151144
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2022.3151144
Author(s): Dominic Rüfenacht
bio
Dominic Rüfenacht leads the science team and is co-founder at Mobius Labs, Berlin, Germany, where he is conducting research on efficient machine learning architectures that aim to understanding image and video content. He received both BSc and MSc in communication systems with specialization in “Signals, Images and Interfaces” from the Swiss Federal Institute of Technology in Lausanne (EPFL), Lausanne, Switzerland. He holds a PhD in electrical engineering from the UNSW Sydney, Sydney, NSW, Australia. His PhD thesis entitled “Novel Motion Anchoring Strategies for Wavelet-Based Highly Scalable Video Compression” won a Springer thesis award and was published as a book. Apart from deep learning for various image and visual analysis tasks, his research interests lie in computational photography and next-generation video compression.
Appu Shaji
bio
Appu Shaji is the chief executive officer and chief scientist at Mobius Labs (http://www.mobiuslabs.com). Prior to founding Mobius Labs, he was the head of research and development at EyeEm, where he led a team of top computer vision researchers from Europe, solving various visual classification and ranking problems. He also founded sight.io, where he developed technology to rate images based on computational aesthetics. sight.io was acquired by EyeEm in March 2014. Shaji has also held postdoctoral positions at the Image and Visual Representation Group (IVRG) and Computer Vision Lab (CVLAB), EPFL. He received a PhD in computer science and engineering from the IIT Bombay, Mumbai, India, for which he received the best PhD thesis award from the Computer Science (CS) department in 2009. His papers have appeared in top computer vision publications including IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) and IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Keyword(s): Facial Expression Analysis, Face Quality Assessment, One-shot Learning, Real-time Video Analysis
Copyright: © 2022 SMPTE

Bibliographic Reference(s)

P. Ekman, “Basic Emotions,” Handbook of Cognition and Emotion, 98:45–60, 1999. EXTERNAL
J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular Margin Loss for Deep Face Recognition,” Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 4690–4699, 2019. EXTERNAL
S. Li and W. Deng, “Deep Facial Expression Recognition: A Survey,” IEEE Trans. Aff. Comput., 2020. EXTERNAL
V. Lepetit, F. Moreno-Noguer, and P. Fua, “Epnp: An Accurate o (n) Solution to the pnp Problem,” Int. J. Comput. Vis., 81(2): 155–166, 2009. identities, allowing for interesting comparisons and statistics on facial expressions. In the figure, we additionally show the appearance rate of specific facial expressions, which is defined as the number of times a facial expression is tagged, normalized by the total number of faces for an identity. As can be seen, both candidates predominantly exhibited an “alpha” face. While Trump showed an exaggerated mouth around 20% of the time, Biden smiled a lot more. Application 2: Facial Expression Summaries We now turn our attention to another application that the proposed system naturally lends itself to, namely the automatic generation of facial expression summaries. In this setting, rather than looking for similar facial expressions, the aim here is to find dissimilar ones. Figure 8 shows an example facial expression summary where we extracted faces for Jim Carrey from a range of movies and shows he starred in. To group similar facial expressions, we use agglomerative clustering (with N = 10 clusters, using ward linkage and Euclidean distance) on facial embeddings extracted from all faces recognized as Jim Carrey. In the figure, we show one representative face for each of the ten clusters. As can be seen, most of the facial expressions shown here are very different from the ones exhibited in the six basic emotions and hence systems trained on basic emotions would not be able to distinguish them in such detail. Furthermore, the above example also reinforces the importance of being able to search facial expressions via a “reference” face, as it would be very hard to give a name/tag to most of these faces. Conclusion We have presented a system that can classify custom facial expressions in videos. Unlike prior art that is limited to a small set of emotions, the proposed system can be used to find custom facial expressions without the need of huge amounts of training data. We show that one “reference face” that portrays the desired facial expression is sufficient to reliably find the facial EXTERNAL
B. Martinez, M. F. Valstar, B. Jiang, and M. Pantic, “Automatic Analysis of Facial Actions: A Survey,” IEEE Trans. Aff. Comput., 10(3):325–347, 2017. EXTERNAL
P. Ekman and H. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychology Press: Palo Alto, CA, 1978. EXTERNAL
G. Chechik, V. Sharma, U. Shalit, and S. Bengio, “Large Scale Online Learning of Image Similarity Through Ranking,” J. Mach. Learn. Res. (JMLR) 11:1109–1135, Mar. 2010. EXTERNAL
M. Tan and Q. V. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” Proc. 36th Int. Conf. Mach. Learn. (ICML) 97:6105–6114, 2019. EXTERNAL
R. Vemulapalli and A. Agarwala, “A Compact Embedding for Facial Expression Similarity,” IEEE Con. Comput. Vision Pattern Recognit. (CVPR), pp. 5683–5692, 2019. EXTERNAL
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases With Noise,” Proc. 2nd Int. Conf. Knowl. Discovery Data Mining, pp. 226–231, 1996. EXTERNAL
J. Deng, J. Guo, Y. Zhou, J. Yu, I. Kotsia, and S. Zafeiriou, “RetinaFace: Single-Stage Dense Face Localisation in the Wild,” CoRR, pp. 5203–5212, 2019. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2022.3151144.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Dominic Rüfenacht and Appu Shaji; Customized Facial Expression Analysis in Video, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3151144

Snippet:

Dominic Rüfenacht and Appu Shaji; Customized Facial Expression Analysis in Video, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3151144

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Dominic Rüfenacht and Appu Shaji; <cite>Customized Facial Expression Analysis in Video</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at <a href="https://doi.org/10.5594/JMI.2022.3151144" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2022.3151144</a></span>

SMPTE's HTML Pub

Preview:

Dominic Rüfenacht and Appu Shaji; Customized Facial Expression Analysis in Video, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022
doi: 10.5594/JMI.2022.3151144
url: https://doi.org/10.5594/JMI.2022.3151144

Snippet:

<li>
Dominic Rüfenacht and Appu Shaji; <cite id="bib-10-5594-jmi-2022-3151144">Customized Facial Expression Analysis in Video</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022
<span class="doi">10.5594/JMI.2022.3151144</span>
</li>