article

Facial expression recognition based on Local Binary Patterns: A comprehensive study

Authors:

Peter W. McOwanAuthors Info & Claims

Image and Vision Computing, Volume 27, Issue 6

Pages 803 - 816

https://doi.org/10.1016/j.imavis.2008.08.005

Published: 01 May 2009 Publication History

Abstract

Automatic facial expression analysis is an interesting and challenging problem, and impacts important applications in many areas such as human-computer interaction and data-driven animation. Deriving an effective facial representation from original face images is a vital step for successful facial expression recognition. In this paper, we empirically evaluate facial representation based on statistical local features, Local Binary Patterns, for person-independent facial expression recognition. Different machine learning methods are systematically examined on several databases. Extensive experiments illustrate that LBP features are effective and efficient for facial expression recognition. We further formulate Boosted-LBP to extract the most discriminant LBP features, and the best recognition performance is obtained by using Support Vector Machine classifiers with Boosted-LBP features. Moreover, we investigate LBP features for low-resolution facial expression recognition, which is a critical problem but seldom addressed in the existing work. We observe in our experiments that LBP features perform stably and robustly over a useful range of low resolutions of face images, and yield promising performance in compressed low-resolution video sequences captured in real-world environments.

References

[1]

Pantic, M. and Rothkrantz, L., Automatic analysis of facial expressions: the state of art. IEEE Transactions on Pattern Analysis and Machine Intelligence. v22 i12. 1424-1445.

[2]

Fasel, B. and Luettin, J., Automatic facial expression analysis: a survey. Pattern Recognition. v36. 259-275.

[3]

M. Pantic, L. Rothkrantz, Toward an affect-sensitive multimodal human-computer interaction, in: Proceeding of the IEEE, vol. 91, 2003, pp. 1370-1390.

[4]

Tian, Y., Kanade, T. and Cohn, J., Handbook of Face Recognition. 2005. Springer.

[5]

Yacoob, Y. and Davis, L.S., Recognizing human facial expression from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence. v18 i6. 636-642.

[6]

Essa, I. and Pentland, A., Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence. v19 i7. 757-763.

[7]

Lyons, M.J., Budynek, J. and Akamatsu, S., Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence. v21 i12. 1357-1362.

[8]

Donato, G., Bartlett, M., Hager, J., Ekman, P. and Sejnowski, T., Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence. v21 i10. 974-989.

[9]

Pantic, M. and Rothkrantz, L., Expert system for automatic analysis of facial expression. Image and Vision Computing. v18 i11. 881-905.

[10]

Tian, Y., Kanade, T. and Cohn, J., Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence. v23 i2. 97-115.

[11]

Cohen, I., Sebe, N., Garg, A., Chen, L. and Huang, T.S., Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and Image Understanding. v91. 160-187.

Digital Library

[12]

L. Yin, J. Loi, W. Xiong, Facial expression representation and recognition based on texture augmentation and topographic masking, in: ACM Multimedia, 2004.

[13]

M. Yeasin, B. Bullot, R. Sharma, From facial expression to level of interests: a spatio-temporal approach, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.

[14]

J. Hoey, J.J. Little, Value directed learning of gestures and facial displays, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.

[15]

Y. Chang, C. Hu, M. Turk, Probabilistic expression analysis on manifolds, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.

[16]

R.E. Kaliouby, P. Robinson, Real-time inference of complex mental states from facial expressions and head gestures, in: IEEE CVPR Workshop on Real-time Vision for Human-Computer Interaction, 2004.

[17]

Pantic, M. and Rothkrantz, L.J.M., Facial action recognition for facial expression analysis from static face images. IEEE Transactions on Systems, Man, and Cybernetics. v34 i3. 1449-1461.

[18]

Zhang, Y. and Ji, Q., Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence. v27 i5. 1-16.

[19]

M.S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, J. Movellan, Recognizing facial expression: machine learning and application to spotaneous behavior, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005.

[20]

F. Dornaika, F. Davoine, Simultaneous facial action tracking and expression recognition using a particle filter, in: IEEE International Conference on Computer Vision (ICCV), 2005.

[21]

C.S. Lee, A. Elgammal, Facial expression analysis using nonlinear decomposable generative models, in: IEEE International Workshop on Analysis and Modeling of Faces and Gestures (AMFG), 2005.

[22]

M. Valstar, I. Patras, M. Pantic, Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data, in: IEEE Conference on Computer Vision and Pattern Recognition Workshop, vol. 3, 2005, pp. 76-84.

[23]

M. Valstar, M. Pantic, Fully automatic facial action unit detection and temporal analysis, in: IEEE Conference on Computer Vision and Pattern Recognition Workshop, 2006, p. 149.

Digital Library

[24]

Pantic, M. and Patras, I., Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics. v36 i2. 433-449.

[25]

Z. Zhang, M.J. Lyons, M. Schuster, S. Akamatsu, Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron, in: IEEE International Conference on Automatic Face & Gesture Recognition (FG), 1998.

[26]

Y. Tian, Evaluation of face resolution for expression analysis, in: CVPR Workshop on Face Processing in Video, 2004.

Digital Library

[27]

Ojala, T., Pietikäinen, M. and Harwood, D., A comparative study of texture measures with classification based on featured distribution. Pattern Recognition. v29 i1. 51-59.

[28]

Ojala, T., Pietikäinen, M. and Mäenpää, T., Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence. v24 i7. 971-987.

[29]

T. Ahonen, A. Hadid, M. Pietikäinen, Face recognition with local binary patterns, in: European Conference on Computer Vision (ECCV), 2004.

[30]

A. Hadid, M. Pietikäinen, T. Ahonen, A discriminative feature space for detecting and recognizing faces, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.

[31]

Feng, X., Hadid, A. and Pietikäinen, M., A coarse-to-fine classification scheme for facial expression recognition. In: International Conference on Image Analysis and Recognition (ICIAR), Lecture Notes in Computer Science, vol. 3212. Springer. pp. 668-675.

[32]

Y. Tian, L. Brown, A. Hampapur, S. Pankanti, A. Senior, R. Bolle, Real world real-time automatic recognition of facial expression, in: IEEE Workshop on Performance Evaluation of Tracking and Surveillance (PETS), Australia, 2003.

[33]

C. Shan, S. Gong, P.W. McOwan, Robust facial expression recognition using local binary patterns, in: IEEE International Conference on Image Processing (ICIP), Genoa, vol. 2, 2005, pp. 370-373.

[34]

S. Liao, W. Fan, C.S. Chung, D.-Y. Yeung, Facial expression recognition using advanced local binary patterns, tsallis entropies and global appearance features, in: IEEE International Conference on Image Processing (ICIP), 2006, pp. 665-668.

[35]

C. Darwin, The Expression of the Emotions in Man and Animals, John Murray, London, 1872.

[36]

M. Suwa, N. Sugie, K. Fujimora, A preliminary note on pattern recognition of human emotional expression, in: International Joint Conference on Pattern Recognition, 1978, pp. 408-410.

[37]

M. Bartlett, G. Littlewort, C. Lainscsek, I. Fasel, J. Movellan, Machine learning methods for fully automatic recognition of facial expressions and facial actions, in: IEEE International Conference on Systems, Man & Cybernetics, Netherlands, 2004.

[38]

M. Turk, A.P. Pentland, Face recognition using eigenfaces, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1991.

[39]

Belhumeur, P.N., Hespanha, J.P. and Kriegman, D.J., Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence. v19 i7. 711-720.

[40]

Bartlett, M.S., Movellan, J.R. and Sejnowski, T.J., Face recognition by independent component analysis. IEEE Transactions on Neural Networks. v13 i6. 1450-1464.

[41]

Shan, C., Gong, S. and McOwan, P.W., Appearance manifold of facial expression. In: Sebe, N., Lew, M.S., Huang, T.S. (Eds.), Vol. 3723 of Lecture Notes in Computer Science, Springer, Beijing. pp. 221-230.

[42]

C. Padgett, G. Cottrell, Representing face images for emotion classification, in: Advances in Neural Information Processing Systems (NIPS), 1997.

[43]

Ekman, P. and Friesen, W., Facial Action Coding System: A Technique for Measurement of Facial Movement. 1978. Consulting Psychologists Press.

[44]

Ekman, P. and Friesen, W., Pictures of Facial Affect. 1976. Consulting Psychologists.

[45]

T. Kanade, J. Cohn, Y. Tian, Comprehensive database for facial expression analysis, in: IEEE International Conference on Automatic Face & Gesture Recognition (FG), 2000.

[46]

P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2001.

[47]

M.R. Everingham, A. Zisserman, Regression and classification approaches to eye localization in face images, in: IEEE International Conference on Automatic Face & Gesture Recognition (FG), 2006, pp. 441-446.

[48]

M. Bartlett, G. Littlewort, I. Fasel, R. Movellan, Real time face detection and facial expression recognition: development and application to human-computer interaction, in: CVPR Workshop on CVPR for HCI, 2003.

[49]

Vapnik, V.N., Statistical Learning Theory. 1998. Wiley, New York.

[50]

C.-W. Hsu, C.-C. Chang, C.-J. Lin, A Practical Guide to Support Vector Classification, Tech. Rep., Taipei, 2003.

[51]

Feng, X., Pietikäinen, M. and Hadid, T., Facial expression recognition with local binary patterns and linear programming. Pattern Recognition and Image Analysis. v15 i2. 546-548.

[52]

G. Guo, C.R. Dyer, Simultaneous feature selection and classifier training via linear programming: a case study for face expression recognition, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2003.

[53]

Schapire, R.E. and Singer, Y., Improved boosting algorithms using confidence-rated predictions. Maching Learning. v37 i3. 297-336.

[54]

G. Zhang, X. Huang, S.Z. Li, Y. Wang, X. Wu, Boosting local binary pattern (lbp)-based face recognition, in: Chinese Conference on Biometric Recognition (SINOBIOMETRICS), 2004, pp. 179-186.

[55]

C. Shan, S. Gong, P.W. McOwan, Conditional mutual information based boosting for facial expression recognition, in: British Machine Vision Conference (BMVC), Oxford, vol. 1, 2005, pp. 399-408.

[56]

Freund, Y. and Schapire, R.E., A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences. v55 i1. 119-139.

[57]

M. Pantic, M. Valstar, R. Rademaker, L. Maat, Web-based database for facial expression analysis, in: IEEE International Conference on Multimedia and Expo (ICME), 2005.

[58]

D. Bolme, M. Teixeria, J. Beveridge, B. Draper, The CSU face identification evaluation system: its purpose, features and structure, in: International Conference on Vision Systems, 2003, pp. 304-311.

[59]

Littlewort, G., Bartlett, M., Fasel, I., Susskind, J. and Movellan, J., Dynamics of facial expression extracted automatically from video. Image and Vision Computing. v24 i6. 615-625.

[60]

Bassili, J.N., Emotion recognition: the role of facial movement and the relative importance of upper and lower area of the face. Journal of Personality and Social Psychology. v37 i11. 2049-2058.

[61]

Zhao, G. and Pietikäinen, M., Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence. v29 i6. 915-928.

Cited By

Liu JLi XZhang JZhai GSu YZhang YWang B(2025)Duration-aware and mode-aware micro-expression spotting for long video sequencesImage Communication10.1016/j.image.2024.117192129:COnline publication date: 7-Jan-2025
https://dl.acm.org/doi/10.1016/j.image.2024.117192
Chen HZhang PGuo CLu KJiang D(2024)Facial Action Unit Representation Based on Self-Supervised Learning With Ensembled Priori ConstraintsIEEE Transactions on Image Processing10.1109/TIP.2024.344625033(5045-5059)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3446250
Zhou HHuang SZhang FXu C(2024)CEPrompt: Cross-Modal Emotion-Aware Prompting for Facial Expression RecognitionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.342477734:11_Part_2(11886-11899)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3424777
Show More Cited By

Facial expression recognition based on Local Binary Patterns: A comprehensive study
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning algorithms

Recommendations

Facial expression recognition based on local binary patterns and local fisher discriminant analysis

Automatic facial expression recognition is an interesting and challenging subject in signal processing, pattern recognition, artificial intelligence, etc. In this paper, a new method of facial expression recognition based on local binary patterns (LBP) ...
Local binary patterns for multi-view facial expression recognition

Research into facial expression recognition has predominantly been applied to face images at frontal view only. Some attempts have been made to produce pose invariant facial expression classifiers. However, most of these attempts have only considered ...
Automatic Facial Expression Recognition Based on Local Binary Patterns of Local Areas
ICIE '09: Proceedings of the 2009 WASE International Conference on Information Engineering - Volume 01

Automatic facial expression recognition is one of the great challenges in facial expression recognition field. An algrithom of automatic facial expression recognition is proposed based on Local Binary Patterns of local areas (LLBP)in this work. First, ...

Comments

Information & Contributors

Information

Published In

cover image Image and Vision Computing

Image and Vision Computing Volume 27, Issue 6

May, 2009

225 pages

ISSN:0262-8856

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2008.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 01 May 2009

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

473
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu JLi XZhang JZhai GSu YZhang YWang B(2025)Duration-aware and mode-aware micro-expression spotting for long video sequencesImage Communication10.1016/j.image.2024.117192129:COnline publication date: 7-Jan-2025
https://dl.acm.org/doi/10.1016/j.image.2024.117192
Chen HZhang PGuo CLu KJiang D(2024)Facial Action Unit Representation Based on Self-Supervised Learning With Ensembled Priori ConstraintsIEEE Transactions on Image Processing10.1109/TIP.2024.344625033(5045-5059)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3446250
Zhou HHuang SZhang FXu C(2024)CEPrompt: Cross-Modal Emotion-Aware Prompting for Facial Expression RecognitionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.342477734:11_Part_2(11886-11899)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3424777
Ye JYu YZheng YLiu YWang Q(2024)Dep-FER: Facial Expression Recognition in Depressed Patients Based on Voluntary Facial Expression MimicryIEEE Transactions on Affective Computing10.1109/TAFFC.2024.337010315:3(1725-1738)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TAFFC.2024.3370103
Li HWang NYang XWang XGao X(2024)Unconstrained Facial Expression Recognition With No-Reference De-Elements LearningIEEE Transactions on Affective Computing10.1109/TAFFC.2023.326388615:1(173-185)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TAFFC.2023.3263886
Yin BYin SLiu CZhang YXi CYin BLing Z(2024)Dynamic facial expression recognition with pseudo‐label guided multi‐modal pre‐trainingIET Computer Vision10.1049/cvi2.1221718:1(33-45)Online publication date: 8-Feb-2024
https://dl.acm.org/doi/10.1049/cvi2.12217
Yang WYu JChen TLiu ZWang XShen J(2024)Multi-threshold deep metric learning for facial expression recognitionPattern Recognition10.1016/j.patcog.2024.110711156:COnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.patcog.2024.110711
Jiang JWang MXiao BHu JDeng W(2024)Joint recognition of basic and compound facial expressions by mining latent soft labelsPattern Recognition10.1016/j.patcog.2023.110173148:COnline publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1016/j.patcog.2023.110173
Chen JShi JXu R(2024)Dual subspace manifold learning based on GCN for intensity-invariant facial expression recognitionPattern Recognition10.1016/j.patcog.2023.110157148:COnline publication date: 17-Apr-2024
https://dl.acm.org/doi/10.1016/j.patcog.2023.110157
Zhu JDing YLiu HChen KLin ZHong W(2024)Emotion knowledge-based fine-grained facial expression recognitionNeurocomputing10.1016/j.neucom.2024.128536610:COnline publication date: 28-Dec-2024
https://dl.acm.org/doi/10.1016/j.neucom.2024.128536
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents