article

Recognition of facial expressions based on salient geometric features and support vector machines

Authors:

Deepak Ghimire,

Sunghwan JeongAuthors Info & Claims

Multimedia Tools and Applications, Volume 76, Issue 6

Pages 7921 - 7946

https://doi.org/10.1007/s11042-016-3428-9

Published: 01 March 2017 Publication History

Abstract

Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.

References

[1]

Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177---188

Digital Library

[2]

Aifanti N, Papachristou A, Delpoulos A (2010) The MUG facial expression database. In Proceeding of 11th international workshop on image analysis for multimedia interactive services, pp 1---4

[3]

Asthana A, Saragih J, Wagner M, Goecke R (2009) Evaluating AAM fitting methods for facial expression recognition. In Proceeding of the international conference on affective computing and intelligent interaction, pp 1---8

[4]

Blome DS (2003) Elastic bunch graph matching. M.Sc. Thesis, Colorado State University: Fort Collins, CO, USA

[5]

Bouguet JY (1999) Pyramidal implementation of the Lucas-Kanade feature tracker. Technological Report, Intel Corporation, Microprocessor Research Lab

[6]

Chang Y, Hu C, Feris R, Turk M (2006) Manifold based analysis of facial expression. Image Vis Comput 24:605---614

Digital Library

[7]

Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machines. Available online: http://www.csie.ntu.edu.tw/~cjlin/libsvm. Accessed on 16 Jan 2015

Digital Library

[8]

Choi HC, Oh SY (2006) Realtime facial expression recognition using active appearance model and multilayer perceptron. In Proceeding of the international joint conference SICE-ICASE, pp 5924---5927

[9]

Cid F, Moreno J, Bustos P, Nunez P (2014) Muecas: a multi-sensor robotic head for affective human robot interaction and imitation. Sensors 14:7711---7737

[10]

Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial emotion recognition. IEEE Trans Affect Comput 5:418---431

[11]

Ekman P (1994) Strong evidence of universal in facial expressions: a reply to Russell's mistaken critique. Psychol Bull 115:268---287

[12]

Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259---275

[13]

Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119---139

Digital Library

[14]

Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13:7714---7734

[15]

Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10:443---458

[16]

Hsu CW, Chang CC, Lin CJ (2010) A practical guide to support vector classification. Technical Report. Department of Computer Science, National Taiwan University, Taiwan

[17]

Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489---501

[18]

Kotisa I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1033---1046

Digital Library

[19]

Kotisa I, Pitas I (2007) Facial expression recognition in image sequence using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172---187

Digital Library

[20]

Li Y, Wang S, Zhao Y, Ji Q (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22:2559---2573

[21]

Liu W, Lu J, Wang Z, Song H (2008) An expression space model for facial expression analysis. In Proceeding of IEEE congress in image and signal processing, pp 680---684

Digital Library

[22]

Mehrabian A (1968) Communication without words. Psychol Today 2:53---56

[23]

Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput Vis Image Underst 115:541---558

Digital Library

[24]

Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22:1424---1445

Digital Library

[25]

Pantic M, Valster R, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Proceeding of IEEE international conference multimedia and expo, pp. 317---321

[26]

Pantic M, Valster M, Rademaker R, Maat L (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specific expression. In Proceeding of 3rd IEEE workshop on CVPR for human communication behavior analysis, pp. 94---101

[27]

Rahulmathavan Y, Phan RCW, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4:83---92

Digital Library

[28]

Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357---1369

Digital Library

[29]

Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact 2014:1---13

Digital Library

[30]

Samal A, Iyenger PA (1994) Automatic recognition of human face and facial expressions: a survey. Pattern Recogn 25:65---77

Digital Library

[31]

Schels M, Schwenker F (2010) A multiple classifier system approach for facial expressions in image sequence utilizing GMM Supervectors. In Proceeding of the 2010 20th international conference on pattern recognition, pp 4251---4254

Digital Library

[32]

Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803---816

Digital Library

[33]

Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682---16713

[34]

Soyel H, Demirel H (2012) Localized discriminative scale invariant feature transform based facial expression recognition. Comput Electr Eng 38:1299---1309

Digital Library

[35]

Sung J, Kin D (2009) Real-time facial expression recognition using STAAM and layered GDA classifiers. Image Vis Comput 27:1313---1325

Digital Library

[36]

Tian Y-L, Kanade T, Cohn JF (2005) Handbook of face recognition. Springer, Berlin, pp 247---275

Digital Library

[37]

Uddin M, Lee J, Kim T (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55:2216---2224

Digital Library

[38]

Uhls YT, Michikyan M, Morris J, Garcia D, Small GW, Zgourou E, Greenfield PM (2014) Five days at outdoor education camp without screen improves pattern skills with nonverbal emotion cues. Comput Hum Behav 39:387---392

Digital Library

[39]

Valster MF, Mehu M, Jiang B, Pantic M, Scherer K (2012) Meta-analysis of the first facial expression recognition challenge. IEEE Trans Syst Man Cybern B Cybern 42:966---979

Digital Library

[40]

Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137---154

Digital Library

[41]

Wiskott L, Fellous JM, Krüger N (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19:775---779

Digital Library

[42]

Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31:210---227

Digital Library

[43]

Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500---508

Digital Library

[44]

Zafeirius S, Pitas I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10:1528---1540

Digital Library

[45]

Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31:39---58

Digital Library

[46]

Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747---3761

[47]

Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29:915---928

Digital Library

[48]

Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11:9573---9588

[49]

Zhi R, Flierl M, Ruan Q, Kleijn WB (2011) Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans Syst Man Cybern B Cybern 41:38---52

Digital Library

[50]

Zhu J, Zou H, Rosset S, Hastie T (2009) Multi-class AdaBoost. Stat Its Interface 2:349---360

Cited By

Nasir MDutta PNandi A(2024)Landmark triangulation-induced Altitude Signature for change detection of human emotion from face image sequenceInnovations in Systems and Software Engineering10.1007/s11334-021-00415-520:2(97-109)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s11334-021-00415-5
Kumar HMartin A(2023)Artificial Emotional IntelligenceExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118651212:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.118651
Shaila SVadivel AAvani S(2023)Emotion estimation from nose feature using pyramid structureMultimedia Tools and Applications10.1007/s11042-023-14682-w82:27(42569-42591)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-14682-w
Show More Cited By

Recognition of facial expressions based on salient geometric features and support vector machines
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Machine learning
    1. Machine learning algorithms

Recommendations

Facial expression recognition based on local region specific features and support vector machines

Facial expressions are one of the most powerful, natural and immediate means for human being to communicate their emotions and intensions. Recognition of facial expression has many applications including human-computer interaction, cognitive science, ...
Facial expression recognition using geometric and appearance features
ICIMCS '12: Proceedings of the 4th International Conference on Internet Multimedia Computing and Service

A novel method using hybrid geometric and appearance features of the difference between the neutral and fully expressive facial expression images is proposed for facial expression recognition in this paper. The difference tends to emphasize the facial ...
Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines

In this paper, two novel methods for facial expression recognition in facial image sequences are presented. The user has to manually place some of Candide grid nodes to face landmarks depicted at the first frame of the image sequence under examination. ...

Comments

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 76, Issue 6

March 2017

1495 pages

ISSN:1380-7501

Issue’s Table of Contents

Copyright © Copyright © 2017 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 March 2017

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nasir MDutta PNandi A(2024)Landmark triangulation-induced Altitude Signature for change detection of human emotion from face image sequenceInnovations in Systems and Software Engineering10.1007/s11334-021-00415-520:2(97-109)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s11334-021-00415-5
Kumar HMartin A(2023)Artificial Emotional IntelligenceExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118651212:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.118651
Shaila SVadivel AAvani S(2023)Emotion estimation from nose feature using pyramid structureMultimedia Tools and Applications10.1007/s11042-023-14682-w82:27(42569-42591)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-14682-w
Hernández-Luquin FEscalante H(2023)Multi-branch deep radial basis function networks for facial emotion recognitionNeural Computing and Applications10.1007/s00521-021-06420-w35:25(18131-18145)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1007/s00521-021-06420-w
Mitsumura KNomiya H(2023)Facial Expression Estimation Using Convolutional Neural Network with Upper and Lower Segmented Face ImagesComputer Information Systems and Industrial Management10.1007/978-3-031-42823-4_31(427-439)Online publication date: 22-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-42823-4_31
Rodríguez Santander MHernández Albarracín JRamírez Rivera A(2021)On the pitfalls of learning with limited dataExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.114991183:COnline publication date: 30-Nov-2021
https://dl.acm.org/doi/10.1016/j.eswa.2021.114991
Ghazouani H(2021)A genetic programming-based feature selection and fusion for facial expression recognitionApplied Soft Computing10.1016/j.asoc.2021.107173103:COnline publication date: 1-May-2021
https://dl.acm.org/doi/10.1016/j.asoc.2021.107173
Nasir MDutta PNandi A(2021)Fuzzy triangulation signature for detection of change in human emotion from face video image sequenceMultimedia Tools and Applications10.1007/s11042-021-11196-180:21-23(31993-32022)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1007/s11042-021-11196-1
Zarbakhsh PDemirel H(2020)4D facial expression recognition using multimodal time series analysis of geometric landmark-based deformationsThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-019-01705-736:5(951-965)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1007/s00371-019-01705-7
Haddad JLezoray OHamel P(2020)3D-CNN for Facial Emotion Recognition in VideosAdvances in Visual Computing10.1007/978-3-030-64559-5_23(298-309)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1007/978-3-030-64559-5_23
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents