RESEARCH

HOME RESEARCH
Behavior Computing
States and Traits
Physiology
Self-assessed Emotion Classification from Acoustic and Physiological Features within Small-group Conversation
Abstract
Individual (personalized) self-assessed emotion recognition has recently received more attention, such as Human-Centered Artificial Intelligence (AI). In most previous studies, researchers utilized the physiological changes and reactions in the body evoked by multimedia stimuli, e.g., video or music, to build a model for recognizing individuals' emotions. However, this elicitation approach is less impractical in the human-human interaction because the conversation is dynamic. In this paper, we firstly investigate the individual emotion recognition task under three-person small group conversations. While predicting personalized emotions from physiological signals is well-studied, few studies focus on emotion classification (e.g., happiness and sadness). Most prior works only focus on binary dimensional emotion recognition or regression, such as valence and arousal. Hence, we formulate the individual emotion recognition task into an individual-level emotion classification. In the proposed method, we consider the physiological changes in each individual's body and acoustic turn-taking dynamics during group conversations for predicting individual emotions. Meanwhile, we assume that the emotional states ofhumans might be affected by the expressive behaviors of other members during group conversations. Also, we hypothesize that people have a higher probability of feeling specific emotions under the related emotional atmosphere. Therefore, we design an ad-hoc technique by simply summing up the Selfassessed emotional annotations of all group members as the group emotional atmosphere (climate) to help the model predict individuals' emotions.We propose a Multi-modal Multi-label Emotion based on Transformer BLSTM at Group Emotional Atmosphere Network (MMETBGEAN) that explicitly considers individual changes and dynamic interaction via physiological and acoustic features during a group conversation integrates group emotional atmosphere information for recognizing individuals' multi-label emotions. We assess the proposed framework on our recently collected extensive Mandarin Chinese collective task group database, NTUBA. The results show that the method outperforms the existing approaches on multi-modal multi-label emotion classification on this database.
Figures
The proposed Multi-modal Multi-label Emotion based on Transformer BLSTM at Group Emotional Atmosphere Network (MMETBGEAN) for individual multi-label emotion classification.
The proposed Multi-modal Multi-label Emotion based on Transformer BLSTM at Group Emotional Atmosphere Network (MMETBGEAN) for individual multi-label emotion classification.
Keywords
Multi-modal interaction | multi-label emotion classification | group emotion regression | group interaction
Authors
Publication Date
2021/10/18
Conference
ICMI
2021 International Conference on Multimodal Interaction
DOI
10.1145/3461615.3485411
Publisher
ACM