Hidden Emotion Detection using Multi-modal Signals

In order to better understand human emotion, we should not recognize only superficial emotions based on facial images, but also analyze so-called inner emotions by considering biological signals such as electroencephalogram (EEG). Recently, several studies to analyze a person’s inner state by using an image signal and an EEG signal together have been reported. However, there have been no studies dealing with the case where the emotions estimated from the image signal and the EEG signal are different, i.e., emotional mismatch. This paper defines a new task to detect hidden emotions, i.e., emotions in a situation where only the EEG signal is activated without the image signal being activated, and proposes a method to effectively detect the hidden emotions. First, when a subject hides the emotion intentionally, the internal and external emotional characteristics of the subject were analyzed from the viewpoint of multimodal signals. Then, based on the analysis, we designed a method of detecting hidden emotions using convolutional neural networks (CNNs) that exhibit powerful cognitive ability. As a result, this study has upgraded the technology of deeply understanding inner emotions. On the other hand, the hidden emotion dataset and source code that we have built ourselves will be officially released for future emotion recognition research.