Exploring the Interplay between Facial Expression Recognition and Physical States

Facial Expression Recognition (FER) systems based on machine learning are trained using diverse databases accessible to the research community.

Some recent works in the field focus on exploring advancements in FER. As outlined in [1], current methodologies commonly used in real-world FER applications reveal a multifaceted approach to understanding and interpreting facial expressions. Within this landscape, Convolutional Neural Networks (CNNs) stand out as a widely employed deep learning methodology in image recognition tasks, emphasizing the importance of the quality of datasets used in various scenarios. Results obtained vary depending on the approach and datasets considered, including Fer2013, CKC, FERC, RAF, DEAP, CASMEI, CASME-I, SAMM, MAHNOB-HCI, Oulu-CASIA, LIRIS-CSE. In previous works [2], it is concluded that the most widely used dataset in FER studies is Fer2013, while DAiSEE is the most utilized emotion dataset at the academic level. Although databases for facial expressions may vary in expression labelling, they consistently reference the seven Ekman's universal expressions [3]. As networks are trained relying on these datasets, researchers have dedicated their efforts to construct models that classify emotions into the seven categories, encompassing the six universally recognized emotions alongside a neutral state. This approach is motivated by two primary factors. Firstly, most of available datasets typically comprise between six to eight labeled expressions, Secondly, it is often more feasible to categorize emotions into a smaller set of classes. Nevertheless, in [4] the authors adopt a different approach by utilizing a dataset that contains avatars performing 32 facial expressions (plus the neutral facial expression). In their work, a simple CNN model is trained on this extensive dataset, therefore they obtain a 33 facial expression classifier. Subsequently, predicted facial expressions are translated to their corresponding emotion (refer to Table 2). Impressively, an accuracy of 0.95 is achieved, a performance level comparable to more complex models currently in use.

A common point in most of the reviewed literature [1], [2], [5] is that it is necessary to transcend mere facial expression recognition towards a more comprehensive understanding of human emotions in diverse scenarios. In this context, physical states like pain, effort and sleepiness, involve facial muscle activity, shaping expressions that aren't straightforwardly linked to emotions. As our current understanding, none of the existing facial expression databases comprehensively address the nuances of various physical states. Existing classifiers focus on recognizing facial expressions attributed to emotional states. Our main goal is to analyse how facial expressions are wrongly reported when the input corresponds to a physical state. To do this, we use a pre-trained FER classifier (only with facial expressions attributed to emotional states) and test it on a new dataset created specifically to include facial expressions associated with physical states. Then, we analyse the results reported by the classifier to understand the implications of lacking classifiers tailored for physical states.

Continuing within the same work setting previously described in Section 2.2, the expressions corresponding to the physical states defined by Gary Faigin have been reproduced. Figure 1 shows the different modeled expressions, except for the Brows down / Perplexed expression, which has been excluded because it does not differ significantly from the Intensity / Attention expression. The difference highlighted by Faigin regarding this expression is the distinct pressure on the eyelids. Unfortunately, this variation cannot be accurately replicated by our synthetic avatars due to the lack of precision and deformers in the ocular region.

The modeled physical states have been defined over 100 avatars created using the Autodesk Character Generator tool [12], deforming the facial meshes according to the muscular activity described by Faigin. The images have been generated within a Unity 3D environment, similar to how UIBVFED has been developed.

The result is a new dataset, UIBVFED Physical States, with 100 synthetic avatars, each reproducing the 16 expressions related to the considered physical states. The dataset is openly accessible on the Zenodo repository, adhering to the FAIR principles¹ .

Una foto de un grupo de personas posando para una foto Descripción generada automáticamente

This work uses a pre-trained CNN model fed with an extension of the original UIBVFED dataset [7]. The used dataset is composed with 100 gender-balanced avatars performing 32 facial expressions (see second column in Table 2). This dataset can be found as a subset of the dataset UIBVFEDPlus-Light [13]. To the best of our knowledge, this CNN model is the only one that classifies the 33 facial expressions attributed to emotional states described by Faigin (see Table 2). Mascaró-Oliver et al. [9] reported a global accuracy of 0.8 when classifying in 33 facial expressions, and a global accuracy of 0.95, when translating the facial expression reported by the model in its associated emotion. The details of the CNN model can be found in [4].

In this work we utilize the pre-trained CNN model on the new UIBVFED Physical States dataset.

In this section, the results returned by the 33-facial expression classifier applied to the images of the UIBVFED Physical States dataset, translated to their associated emotion according to Table 2 and following the procedure described in [9], are presented.

As expected, none of the images has correctly been classified, since the model was trained with images corresponding to a different set of categories, which did not include physical states. Table 3 illustrates how the CNN model classifies the facial expressions corresponding to physical states. These expressions are translated to their associated emotions.

In table 3, the first column displays the physical states expressions, while the last column indicates the expressions with which they could be confused according to Faigin (see Table 2). Columns 2 to 8 show the percentage of images for each physical state classified into facial expressions related to each emotion. These data allow for a proper interpretation of the level of confusion in our model. For instance, the CNN model classifies the Pain 1 expression images mostly as Joy (62%), Anger (13%), and Sadness (11%). The Pain 1 physical expression has a theoretical potential confusion with the Crying, Laughing and Exertion expressions according to Faigin. As Crying relates to the emotion of Sadness, Laughing relates to the emotion of Joy, and Exertion relates to Anger, the behavior of the classification (despite being incorrect relating to physical states) agrees in an 86% with the theoretical confusion behavior reported by Faigin.

The same results can be extracted for each of the physical stages, leading to a “coherent with the literature” classification of the physical states within the six universal emotions. However, these results are inaccurate. Even if it is true that the facial expression of Laughing shares facial features with the facial expressions of Pain, and therefore they can be confused, it is not acceptable to classify a Pain physical state as Joy emotion. Therefore, it is necessary to count on classifiers able to correctly label physical states, besides emotions. For that reason, it is mandatory to create new datasets to train those models.

One of the limitations of the present work is the classification of the physical states expressions. In this study, we adopted Faigin's classification, which has some justified shortcomings. Regarding facial expressions induced by the reproduction of visemes, only the Singing 1 and Singing 2 expressions are considered. Aspects of speech cannot be completely separated from emotions, although its treatment entails complexity beyond the scope of this work. Other physical states expressions may allow for variations that have not been taken into consideration, for example, configurations of Exertion with open eyes or Drowsiness with the eyes completely closed. We understand that, despite the associated computational cost, including all these subtle variations would increase precision.

Another limitation concerns to the realism of the avatars. Our work is constrained by the avatars generated by the Autodesk Character Generator tool and the accompanying deformers. We have encountered shortcomings in defining movements of pressure on the eyelids and in the precise control of the lips. In addition to these deficiencies related to geometric deformation, there are also issues associated with the texture applied to the avatars. Specifically, the facial wrinkles in our avatars are defined by textures applied to the color and bump of the material. Wrinkles are a crucial part of facial expressions, and being detached from geometric deformers implies a lack of control in their manipulation.

Any study of FER based on static images suffers from the inherent limitation of lacking context inherent to an isolated image. A significant improvement would be achieved, not only by defining physical states with video sequences, but also by using videos for the definition of facial expressions associated with emotional states.

In this work, we have highlighted the issues arising from the omission of facial expressions related to physical states in the context of Facial Expression Recognition (FER) tasks. It becomes evident that the datasets used to train recognition systems, lacking labels associated with physical states, inevitably introduce bias into their estimations. Our research offers a quantitative analysis of this bias, comparing the results obtained with the expected interpretations provided by Faigin. While our findings are specific to a simple CNN model, they contribute valuable insights into the distribution of errors, shedding light on potential challenges across other recognition methods.

As a significant contribution, we offer the research community a dataset containing facial expressions of physical states. This novel dataset aims to address and alleviate the issues associated with biased estimations in FER tasks. By including a comprehensive range of physical states, the UIBVFED Physical States dataset facilitates more inclusive and accurate training of recognition systems, enhancing their robustness and applicability across diverse scenarios.