research-article

Image ratio features for facial expression recognition application

Authors:

Mengchu ZhouAuthors Info & Claims

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, Volume 40, Issue 3

Pages 779 - 788

https://doi.org/10.1109/TSMCB.2009.2029076

Published: 01 June 2010 Publication History

Abstract

Video-based facial expression recognition is a challenging problem in computer vision and human-computer interaction. To target this problem, texture features have been extracted and widely used, because they can capture image intensity changes raised by skin deformation. However, existing texture features encounter problems with albedo and lighting variations. To solve both problems, we propose a new texture feature called image ratio features. Compared with previously proposed texture features, e.g., high gradient component features, image ratio features are more robust to albedo and lighting variations. In addition, to further improve facial expression recognition accuracy based on image ratio features, we combine image ratio features with facial animation parameters (FAPs), which describe the geometric motions of facial feature points. The performance evaluation is based on the Carnegie Mellon University Cohn-Kanade database, our own database, and the Japanese Female Facial Expression database. Experimental results show that the proposed image ratio feature is more robust to albedo and lighting variations, and the combination of image ratio features and FAPs outperforms each feature alone. In addition, we study asymmetric facial expressions based on our own facial expression database and demonstrate the superior performance of our combined expression recognition system.

References

[1]

D.-Q. Dai and P. C. Yuen, "Face recognition by regularized discriminant analysis," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 4, pp. 1080-1085, Aug. 2007.

[2]

K.-C. Wong, W.-Y. Lin, Y. H. Hu, N. Boston, and X. Zhang, "Optimal linear combination of facial regions for improving identification performance," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 5, pp. 1138-1148, Oct. 2007.

[3]

R. Singh, M. Vatsa, A. Ross, and A. Noore, "A mosaicing scheme for pose-invariant face recognition," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 37, no. 5, pp. 1212-1225, Oct. 2007.

[4]

H. Zhao and P. C. Yuen, "Incremental linear discriminant analysis for face recognition," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 38, no. 1, pp. 210-221, Feb. 2008.

Digital Library

[5]

P. Ekman and W. V. Friesen, "Constant across cultures in the face and emotion," J. Pers. Soc. Psychol., vol. 17, no. 2, pp. 124-129, 1971.

[6]

G. Guo and C. R. Dyer, "Learning from examples in the small sample case: Face expression recognition," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 35, no. 3, pp. 477-488, Jun. 2005.

Digital Library

[7]

P. Ekman, W. V. Friesen, and J. C. Hager, Facial Action Coding System. Salt Lake City, UT: Human Face, 2002.

[8]

G. Chanel, K. Ansari-Asl, and T. Pun, "Valence-arousal evaluation using physiological signals in an emotion recall paradigm," in Proc. IEEE Int. Conf. Syst., Man, Cybern., 2007, pp. 2662-2667.

[9]

L. Ma and K. Khorasani, "Facial expression recognition using constructive feedforward neural networks," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 34, no. 3, pp. 1588-1595, Jun. 2004.

Digital Library

[10]

G. D. Guo and C. R. Dyer, "Simultaneous feature selection and classifier training via linear programming: A case study for face expression recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., Toronto, ON, Canada, 2003, vol. 1, pp. 346-352.

[11]

K. Anderson and P. W. McOwan, "A real-time automated system for recognition of human facial expressions," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 36, no. 1, pp. 96-105, Feb. 2006.

Digital Library

[12]

H. Wang and N. Ahuja, "Facial expression decomposition," in Proc. IEEE Int. Conf. Comput. Vis., Nice, France, Oct. 2003, vol. 2, pp. 958-965.

[13]

I. Cohn, N. Sebe, L. Chen, A. Garg, and T. S. Huang, "Facial expression recognition from video sequences: Temporal and static modeling," Comput. Vis. Image Underst., vol. 91, no. 1/2, pp. 160-187, Jul. 2003.

Digital Library

[14]

Y. Zhang and Q. Ji, "Facial expression understanding in image sequences using dynamic and active visual information fusion," in Proc. IEEE Int. Conf. Comput. Vis., Nice, France, Oct. 2003, vol. 2, pp. 113-118.

[15]

N. Sebe, M. S. Lew, I. Cohen, Y. Sun, T. Gevers, and T. S. Huang, "Authentic facial expression analysis," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., Seoul, Korea, May 2004, pp. 517-522.

Digital Library

[16]

H. Seyedarabi, A. Aghagolzadeh, and S. Khanmohammadi, "Recognition of six basic facial expressions by feature-points tracking using RBF neural network and fuzzy inference system," in Proc. IEEE Int. Conf. Multimed. Expo, Taipei, Taiwan, Jun. 2004, pp. 1219-1222.

[17]

Z. Zhu and Q. Ji, "Robust real-time face pose and facial expression recovery," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., New York, 2006, vol. 1, pp. 681-688.

[18]

Y. L. Tian, T. Kanade, and J. F. Cohn, "Facial expression analysis," in Handbook of Face Recognition, S. Z. Li and A.K. Jain, Eds. New York: Springer-Verlag, 2005, pp. 247-276.

[19]

J. J. Lien, T. Kanade, J. F. Cohn, and C.-C. Li, "Subtly different facial expression recognition and expression intensity estimation," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., Santa Barbara, CA, 1998, pp. 853-859.

[20]

Y. Gao, S. H. Leung, and H. Leung, "Facial expression recognition from line-based caricatures," IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 33, no. 3, pp. 407-412, May 2003.

Digital Library

[21]

Y. Zhang and Q. Ji, "Active and dynamic information fusion for facial expression understanding from image sequences," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 5, pp. 699-714, May 2005.

Digital Library

[22]

Y. Tian, T. Kanade, and J. F. Cohn, "Recognizing action units for facial expression analysis," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 2, pp. 97-115, Feb. 2001.

Digital Library

[23]

A. M. Martínez, "Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 6, pp. 748-763, Jun. 2002.

Digital Library

[24]

J. Han and K.-K. Ma, "Rotation-invariant and scale-invariant Gabor features for texture image retrieval," Image Vis. Comput., vol. 25, no. 9, pp. 1474-1481, Sep. 2007.

Digital Library

[25]

G. Zhao and M. Pietikäinen, "Dynamic texture recognition using local binary patterns with an application to facial expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 915-928, Jun. 2007.

Digital Library

[26]

J. Shi, A. Samal, and D. Marx, "How effective are landmarks and their geometry for face recognition?" Comput. Vis. Image Underst., vol. 102, no. 2, pp. 117-133, May 2006.

Digital Library

[27]

Y. Chang, C. Hu, R. Feris, and M. Turk, "Manifold based analysis of facial expression," Image Vis. Comput., vol. 24, no. 6, pp. 506-614, Jun. 2006.

Digital Library

[28]

T. Kanade, J. F. Cohn, and Y. Tian, CMU-Pittsburgh AU-Coded Face Expression Image Database. {Online}. Available: http://vasc.ri.cmu.edu/ idb/html/face/facial_expression/

[29]

M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, "Coding facial expressions with Gabor wavelets," in Proc. IEEE Int. Conf. Autom. Face Gesture Recog., Nara, Japan, 1998, pp. 200-205.

[30]

International Organization for Standardization, Facial Animation Parameters, 1998. Text for ISO/IEC FDIS 14496-2 Visual, ISO/IEC JTC1/SC29/ WG11 N2502.

[31]

Z. Liu, Y. Shan, and Z. Zhang, "Expressive expression mapping with ratio images," in Proc. ACM SIGGRAPH, Los Angeles, CA, 2001, pp. 271-276.

[32]

J. Bu, M. Song, Q. Wu, and C. Chen, "Sketch based facial expression recognition using graphics hardware," in Proc. Int. Conf. Affective Comput. Intell. Interaction, Beijing, China, 2005, vol. 3784, pp. 72-80.

Digital Library

[33]

M. Covell and C. Bregler, "Eigen-points," in Proc. IEEE Int. Conf. Image Process., Laussane, Switzerland, 1996, pp. 471-474.

[34]

T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Active shape models--Their training and application," Comput. Vis. Image Underst., vol. 61, no. 1, pp. 38-59, Jan. 1995.

Digital Library

[35]

T. F. Cootes, G. Edwards, and C. J. Taylor, "Comparing active shape models with active appearance models," in Proc. Brit. Mach. Vis. Conf., Nottingham, U.K., 1999, pp. 173-182.

[36]

F. Lavagetto and R. Pockaj, "The facial animation engine: Towards a high-level interface for the design of MPEG-4 compliant animated faces," IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 2, pp. 277-289, Mar. 1999.

Digital Library

[37]

D. Cai, X. He, Y. Hu, J. Han, and T. Huang, "Learning a spatially smooth subspace for face recognition," in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recog., Minneapolis, MN, 2007, pp. 1-7.

[38]

D. Cai, X. He, X.Wang, H. Bao, and J. Han, "Locality preserving nonnegative matrix factorization," in Proc. Int. Joint Conf. Artif. Intell., Pasadena, CA, 2009, pp. 1010-1015.

[39]

X. He, M. Ji, and H. Bao, "Graph embedding with constraints," in Proc. Int. Joint Conf. Artif. Intell., Pasadena, CA, 2009, pp. 1065-1070.

[40]

M. F. Valstar and M. Pantic, "Biologically vs. logic inspired encoding of facial actions and emotions in video," in Proc. IEEE Int. Conf. Multimed. Expo, Toronto, ON, Canada, 2006, pp. 325-328.

[41]

P. Sungsoo, S. Jongju, and K. Daijin, "Facial expression analysis with facial expression deformation," in Proc. IAPR Int. Conf. Pattern Recog., Tampa, FL, 2008, pp. 1-4.

[42]

M. Valstar, M. Pantic, and I. Patras, "Motion history for facial action detection in video," in Proc. IEEE Int. Conf. Syst., Man, Cybern., Hague, The Netherland, 2004, pp. 635-640.

Cited By

Barman ADutta P(2024)Facial expression recognition using Reversible Neural NetworkApplied Soft Computing10.1016/j.asoc.2024.111815162:COnline publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.asoc.2024.111815
Sima YYi JChen AJin Z(2021)Automatic expression recognition of face image sequence based on key-frame generation and differential emotion featureApplied Soft Computing10.1016/j.asoc.2021.108029113:PAOnline publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1016/j.asoc.2021.108029
Testa RCorrêa CMachado-Lima ANunes F(2019)Synthesis of Facial Expressions in PhotographsACM Computing Surveys10.1145/329265251:6(1-35)Online publication date: 28-Jan-2019
https://dl.acm.org/doi/10.1145/3292652
Show More Cited By

Image ratio features for facial expression recognition application
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks

Recommendations

A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In ...
Person-Similarity Weighted Feature for Expression Recognition
Computer Vision – ACCV 2007
Abstract
In this paper, a new method to extract person-independent expression feature based on HOSVD (Higher-Order Singular Value Decomposition) is proposed for facial expression recognition. With the assumption that similar persons have similar facial ...
Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware
Abstract
Since the distinction between two expressions is fairly vague, usually a subtle change in one part of the human face is enough to change a facial expression. Most of the existing facial expression recognition algorithms are not robust enough ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics Volume 40, Issue 3

Special issue on game theory

June 2010

425 pages

ISSN:1083-4419

Issue’s Table of Contents

Copyright © 2010.

Publisher

IEEE Press

Publication History

Published: 01 June 2010

Revised: 29 April 2009

Received: 18 December 2008

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Barman ADutta P(2024)Facial expression recognition using Reversible Neural NetworkApplied Soft Computing10.1016/j.asoc.2024.111815162:COnline publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.asoc.2024.111815
Sima YYi JChen AJin Z(2021)Automatic expression recognition of face image sequence based on key-frame generation and differential emotion featureApplied Soft Computing10.1016/j.asoc.2021.108029113:PAOnline publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1016/j.asoc.2021.108029
Testa RCorrêa CMachado-Lima ANunes F(2019)Synthesis of Facial Expressions in PhotographsACM Computing Surveys10.1145/329265251:6(1-35)Online publication date: 28-Jan-2019
https://dl.acm.org/doi/10.1145/3292652
Barman ADutta P(2019)Facial expression recognition using distance and texture signature relevant featuresApplied Soft Computing10.1016/j.asoc.2019.01.01177:C(88-105)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1016/j.asoc.2019.01.011
Vishnu Priya R(2019)Emotion recognition from geometric fuzzy membership functionsMultimedia Tools and Applications10.1007/s11042-018-6954-978:13(17847-17878)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1007/s11042-018-6954-9
Kumar SSingh SKumar J(2018)Live Detection of Face Using Machine Learning with Multi-feature MethodWireless Personal Communications: An International Journal10.1007/s11277-018-5913-0103:3(2353-2375)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.1007/s11277-018-5913-0
Peng YYin H(2018)Facial expression analysis and expression-invariant face recognition by manifold-based synthesisMachine Vision and Applications10.1007/s00138-017-0895-629:2(263-284)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1007/s00138-017-0895-6
Chen XZhou ZZhang JLiu ZHuang Q(2016)Local convex-and-concave patternInformation Sciences: an International Journal10.1016/j.ins.2016.05.017363:C(120-139)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1016/j.ins.2016.05.017
Zhang LTjondronegoro DChandran VEggink J(2016)Towards robust automatic affective classification of images using facial expressions for practical applicationsMultimedia Tools and Applications10.1007/s11042-015-2497-575:8(4669-4695)Online publication date: 1-Apr-2016
https://dl.acm.org/doi/10.1007/s11042-015-2497-5
Happy SRoutray A(2015)Automatic facial expression recognition using features of salient facial patchesIEEE Transactions on Affective Computing10.1109/TAFFC.2014.23863346:1(1-12)Online publication date: 25-Feb-2015
https://dl.acm.org/doi/10.1109/TAFFC.2014.2386334
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents