ABSTRACT:
OBJECTIVE: Emotions affect health and health affects emotions. When properly recognized and interpreted,
emotions can aid in the prevention, diagnosis, and treatment of many diseases. Affective
computing, the use of computer technology to detect one or more signals associated
with human emotions, is a promising field. However, research into the use of emotion
recognition in healthcare remains scarce. It is crucial to explore the reasons behind
this phenomenon and identify neglected methods with potential for human health. We
review methods and technologies used in emotion recognition and their applications
in healthcare, highlighting methods not discussed in previous reviews, including electroencephalography
and electrocardiography, thermal imaging, bracelets, skin conductance, and audio.
METHODS: A metric based on reproducibility and population was established to assess the quality
of included articles. Based on the metrics established, we surveyed and analyzed studies
in which affective computing tools were applied to health, to qualify and identify
the challenges of the area.
RESULTS: We found many challenges to be overcome in detecting and recognizing human emotions,
related to sample size, low study quality, and reproducibility issues. We list and
discuss the main current challenges, ways to overcome them, and perspectives for the
future, focusing on the application of affective technologies in healthcare and the
establishment of a gold standard.
CONCLUSION: Three suggestions are proposed: 1) to conduct studies focused on obtaining a gold
standard; 2) to conduct studies with larger sample sizes, greater diversity, and in
less controlled environments, using replicable methodologies and making data and methods
available; and 3) to further explore the potential use of emotion detection in healthcare.
Keywords: Emotion; sensors; technology; health; recognition.
FIGURES
| Note: How to cite this article: Lima ES, Perico CP, Nichio BTL, Linhares GT, Schwanka A, Silveira RD, et al. Emotional recognition technologies applied to health: review and challenges. Braz J Psychiatry. 2025;47:e20243963. http://doi.org/10.47626/1516-4446-2024-3963 |
| Received: April 19 2021; Accepted: August 16 2021 |
INTRODUCTION
Emotion and health are a two-way street.1 Human emotion, mediated by neurotransmitters and other molecules involved in the physiological and neural triggering of emotions, affects the body's homeostasis. In the same way, imbalances in homeostasis trigger emotions. Loss of homeostasis is strongly linked to the development of illnesses, whether mental (such as depression and anxiety) or physiological (such as irritable bowel syndrome [IBS]). Lima et al.1 provides more details on the relationship between emotions and health as a two-way street, with a focus on older adults and those affected by coronavirus disease 2019 (COVID-19).
Since ancient Greece, philosophers and scientists such as Socrates and Hippocrates have considered emotions as determinants of human health and disease.2 In the 20th century, from the 1970s onwards, biochemistry presented evidence of the relationship between emotions and health. One such discovery was the interaction between molecules and opioid receptors throughout the endocrine, nervous, and digestive systems. This interaction produces physiological effects in the body that affect mood.3 Further biochemical evidence was provided by the discovery of communication between the hypothalamus-pituitary and the immune system, linking emotions to disease.4
Over the past 20 years, the growth of research on the relationship between emotions and health has led to the emergence of the term affective computing, coined by Picard.5 Affective computing is an incipient field that involves the development of algorithms designed to detect one or more signals associated with human emotions. Due to its recent emergence, there are still few review studies available. Among the existing methods, the most affordable and least invasive are the least explored, despite their greater practical applicability and coherence, as they do not cause changes in emotional states due to their low invasiveness.
Within this context, we conducted a review focusing on affective computing and current technologies for the detection and recognition of human emotions as applied to human health. Facing a wide diversity of emotion detection methods and studies - without necessarily having a link to health - we first aim to establish whether current methods are suitable to be applied to human health. We then determine which obstacles and challenges exist that hamper these advances. We focus on the emotion recognition methods less explored in previous studies, such as those of Lunesk et al.2 and Dzedzickis et al.,6 since these are neglected methods in the literature.
In the Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes section, we present a survey of existing applications of emotion recognition in human health and potential uses, grouped by an application regardless of the technology or combination of methods employed. In the Affective computing tools: monomodal and multimodal methods for emotion detection section, we review technologies - initially, by individual signal type (Monomodal studies in emotion detection: tools and techniques for physiological signal analysis), then the combination of signals (Emotion recognition using multimodal methods: a decade of research), and finally the challenges and solutions of real-world implementation (Recognizing emotions in the real world). Finally, in the Discussion and challenges section, we present our considerations and the challenges identified.
BACKGROUND
Emotions are a category of expressions associated with a biological and/or neural basis. An emotion is a response to an internal or external stimulus. This stimulus is signaled to the brain, which causes behavioral and physiological reactions.7 Faced with an emotional stimulus, the brain releases substances that generate physiological responses that affect the functional balance of the organism.8
Studies in the field of affective neuroscience have revealed an interactive signaling network whereby the peripheral nervous system and the central nervous system receive and generate the experience of emotion.9 This experience activates a brain-body connection10 through substances (neurotransmitters, steroids, and peptides) called emotion molecules,3 which stimulate sympathetic and parasympathetic pathways, shaping physiological activity to respond to constant changes in the internal and external environment.11 The result of this variation translates to homeostatic alterations at the moment of an emotional reaction.
Devising algorithms to detect the various signals potentially associated with different human emotions is a complex task. Emotions can be captured by different signals, including temperature, voice variation, electrical signals, etc., and this may be done by measuring one (monomodal) or more (multimodal) such signals.
The concept of affective computing was developed by Picard.12 This line of research has emerged to provide reliable bases for studies that link emotions and computers, through the use of theoretical descriptions of human affective states, as well as their influence on social, cognitive, physical, or other levels of human actions.2 It focuses on sensors and algorithms whose aim is to recognize and communicate emotional responses.13 Detecting, processing, and interpreting emotional responses by means of emotional expression has various medical purposes,2 and constitutes an objective, safe, and efficient way of using emotional information to support well-being and quality of life.14
Technological methods for classifying emotions require models to perform the categorization. Many papers have sought to provide an overview of emotion categorization models.15-20 There are three general approaches to modeling and classifying emotions: discrete, dimensional, and evaluative. The most widely used and accepted in the literature are the discrete, particularly Ekman's model of six basic emotions, and the dimensional, especially the arousal-valence model.
The discrete model has its origins in studies focused on facial expressions21 and is linked to evolutionary stimuli and triggers. Studies have already linked discrete emotions, called basic emotions, to neural structures.22 Researchers using this theory focus on identifying the biological and/or neural basis of emotions.19 The most widely used sets of basic emotions in the literature correspond to the six basic emotions known as "Ekman's big six": anger, surprise, disgust, enjoyment, fear, and sadness.21 This list is based on studies that involved a survey of facial expressions and present evidence for the existence of universal expressions that are cross-cultural, evidencing that these emotions are "innate" rather than culturally acquired or learned.
In the dimensional model, emotion is understood as a cultural process, learned and expressed differently in each culture. Researchers in this line are more interested in how feelings diverge culturally and how their interpretations vary from subject to subject in different situations. In this theory, infinite affective states exist in a multidimensional space.19 The valence-arousal15 and the approach-avoidance axes are among the leading dimensional models.
As noted above, emotional states can be interpreted monomodally, on the basis of only one signal, or multimodally, combining several parameters.2,12-14,23,24 Affective computing tools capture these signals and classify emotions25 by applying machine learning techniques26 such as classification, regression, unsupervised algorithms, and deep learning. Classification models are trained to assign an emotional category (such as happiness, sadness, anger, among others) to input data. Regression can be used to predict continuous scores related to emotion intensity. Unsupervised algorithms, such as clustering, can be employed to automatically group data into emotional categories without prior labels. Recurrent neural networks can be applied to temporal sequences, such as sentiment analysis of speech or text over time.27 -30 In particular, deep learning involving the use of deep neural networks to automatically learn high-level representations from raw data can automatically extract features from data, eliminating the need for manual extraction.31
Multimodal approaches combine information from different sources to enhance accuracy and have been widely utilized in artificial intelligence (AI) approaches.14 They usually employ analysis of audio signals, such as prosody in speech (pitch, rhythm, intonation), and natural language processing (NLP) techniques for processing written text and extracting relevant information.32,33 Model performance is evaluated using metrics such as accuracy, F1-score, and classification correlation, depending on the specific problem. In this context, specific datasets exist for emotion identification, such as the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) for audio and AffectNet for facial images.34,35
METHODS
The objective of this review is to conduct a comprehensive analysis of affective computing tools applied in health care. Initially, we conducted a thorough investigation of studies focusing on technologies for emotion recognition. Subsequently, we systematically surveyed and analyzed various studies that have employed affective computing tools in the context of health care.
Technologies employed in emotion recognition
We performed our analysis according to the steps described in Figure 1. First, we searched the PubMed and Google Scholar engines for the terms "affective computing" and "affective technologies." Considering previous reviews on the topic2,6 and a previous PubMed search, presented in Supplementary Table S1, we focused on articles using the following methods of obtaining signals: 1) electrocardiography (ECG); 2) electroencephalography (EEG); 3) galvanic skin response (GSR)/skin conductance (SC); 4) thermal camera (TC); 5) bracelet; and 6) multimodal methods.

We acknowledge the significance of multimodal analyses and their potential to improve the precision of emotional recognition outcomes. However, our investigation focused specifically on the methodologies employed in monomodal research, allowing for a more targeted evaluation of methodological rigor without the confounding complexities introduced by integrating diverse device outputs in multimodal studies. This deliberate emphasis sought to circumvent methodological redundancies within individual studies, particularly given the heterogeneity of devices employed to capture analogous physiological signals. As a result, we selected only articles on monomodal techniques and evaluated their reproducibility and analytical quality using the framework proposed by Peng36 and our metrics, respectively.
Through the search results, we considered mainly, but not only, studies on Ekman's basic emotions (joy/happiness, fear, anger, sadness, disgust, and surprise).37 Of these, we selected 104 monomodal and multimodal articles. As already noted, the monomodal method analyses emotional event signals using a single input,38 while the multimodal method uses more than one source for emotion detection and recognition.39 Among these articles, we separated for analysis only those on monomodal techniques, comprising 61.54% of the studies: EEG (41.66%), ECG (26.66%), TC (16.66%), bracelet (8.33%), and GSR/SC (6.66%).
In this study, we considered three classifications for methods in terms of their invasiveness. Non-invasive methods are those in which the participant does not realize that they are being analyzed or monitored (e.g., video, audio). Low-invasive methods are those that still allow participants to remain comfortable with minimal stressors (ECG, EEG, and GSR). Invasive methods, which were not included in this study, are highly stressful and can interfere with emotional responses (computed tomography and electrocorticography).
Next, we analyzed the selected articles based on reproducibility and the number of participants. Reproducibility of results is the standard by which scientific claims are evaluated.36 The number of participants, or sample size, confers reliability to the study. The sample size corresponds statistically to the population and should be statistically significant.40
In many fields of study, there are examples of scientific research that cannot be fully replicated due to a lack of resources to enable its reproduction. In this sense, Peng36 has proposed major criteria to determine the reproducibility of a study ensuring scientific validation for its results. We assessed the reproducibility of the reviewed articles with a metric inspired by Peng.36,41
We considered the following items for evaluating the articles surveyed: number of participants; Peng's metric, which consists of a discrete metric that awards 1 point for the following requirements: provide (i) data, (ii) the code used, (iii) and the link between data and code36; and the result of the metric R, obtained as follows:
R = l o g n + p (1)where n is the number of participants and p reproducibility (Peng's metric). The higher the value of R, the better rated the article. Finally, we present our results by ranking the articles in order of score.
An additional meta-analysis was carried out, using 64 articles on monomodal techniques, to determine which signal was best for each type of discrete emotion (considering only Ekman's big six). Due to the availability of statistical data, the accuracy metric was used to compare the studies. The highest average accuracy was used as a criterion for comparing the different monomodal signals.
Affective computing applied to health
We performed our analysis according to the steps illustrated in Figure 2. We searched for the terms "alexithymia affective computing," "affective computing pain," "neurodegenerative diseases affective computing," "affective computing disorder of consciousness," and "depression affective computing" on PubMed and Google Scholar (Supplementary Table S2). The search was limited to the period from 2010 to 2023. In view of the high number of results presented by the Google Scholar platform, we applied filtering using advanced search criteria: exact phrase and the use of at least one of the words defined in the search query. Use of these filters yielded a median reduction of 98% in the number of results, as shown in Supplementary Table S3.

We did not survey duplicate results present in both platforms. However, we did find a double result for neurodegenerative diseases and consciousness disorder, as described in the Impact of affective computing in health: gaps and insights from current research section.
When searching, we realized that the results for stress and depression would need to be refined further, due to the large number of articles not related to affective technologies being returned. Depression and stress per se are commonly related to affective states. Thus, we added the term "emotion" for a more targeted search that would yield studies focusing on emotions.
After the search, we selected articles for our analysis based on the titles and abstracts. Review articles, books, and research based on purely subjective data, such as questionnaires and self-assessments, were excluded. Next, we analyzed, based on our metrics, the studies that used affective computing techniques applied to health.
AFFECTIVE COMPUTING TOOLS: MONOMODAL AND MULTIMODAL METHODS FOR EMOTION DETECTION
Affective computing tools applied to health are based on particular responses or signals of the human body. One of the most common such signals is the heartbeat, captured by methods such as wristbands,42 gloves with sensors,43 or devices attached directly to the chest.44 Brain signals are usually captured using classical methods such as EEG45 or functional magnetic resonance imaging (fMRI).46 The skin conductivity reading method uses the mineral salts expelled by sweat, which conduct electricity, to identify variations in this parameter caused by emotions.47 In addition to these methods, those focused on capturing facial expressions are also widely studied. Pantic & Bartlett,48 for example, reviewed techniques for identifying facial expressions over a decade. In addition to broad representation in the literature, these methods can rely on immense datasets, such as EmotioNet, which contains one million facial expression images collected from the Internet.49 So much data requires specific studies.
Supplementary Figure S1 presents the devices and methods used to identify emotions explored in this review. The following section describes the main signals captured by each technology.
Monomodal studies in emotion detection: tools and techniques for physiological signal analysis
As noted above, our review was limited to studies using monomodal methods. Within these, we focused on five specific tools: ECG, EEG, SC, TC, and wearable devices (smart bracelets or wristbands).
Electroencephalography
The EEG records electrical activity in the brain, reflecting neurophysiological signals from various brain regions involved in perception and cognition. For instance, the frontal lobe is associated with thought and consciousness, the temporal lobe processes sensory stimuli such as smells and sounds, the parietal lobe integrates sensory information and manages object control, and the occipital lobe is related to vision.50 Fear, for example, activates the amygdala, a region essential for emotional processing51 which plays a role in various behaviors.52-54 In EEG studies, the signal-to-noise ratio (SNR) is a key metric, representing the relationship between signal power and noise power.55 Researchers typically examine the frequency, energy, and power spectra of different EEG bands.56 Some authors define five rhythmic frequency bands, while others consider six, with minor variations in their thresholds. The five main bands are: delta (0.5-3 Hz), related to deep sleep and exhaustion; theta (4-8 Hz), occurring during sleep and meditation; alpha (9-13 Hz), present in calm and unstressed states; beta (14-30 Hz), associated with active thought and work; and gamma (above 31 Hz), linked to focus, anxiety, and emotional stress.57,58 Studies on basic emotions have associated the gamma and alpha bands with happiness and sadness,59-61 while fear has been connected to delta, theta, alpha, and gamma waves in various brain regions.62,63 Continuous emotional models, such as valence-arousal, are more commonly related to EEG signals than discrete models.57,58 (For a comprehensive review of EEG and emotion detection, see Liu et al.57 or Rahman et al.58) One limitation of EEG is the presence of noise, common in electrical signal readings, which poses challenges for implementing machine learning algorithms in emotion recognition. Additionally, the signal acquisition process often requires tens to hundreds of electrodes and sampling periods lasting several seconds. This results in datasets with relatively few samples, which has led to a dearth of EEG data on cognitive neuroscience tasks.55
Electrocardiography
The emotional experience, triggered in the brain, leads to changes in heart rate through the autonomic nervous system (ANS). These changes can be detected via ECG, which captures electrical signals from the heart.24 ECG can also measure heart rate variability (HRV), a non-invasive marker of ANS activity.64 The ECG has been widely used in affective computing to assess heart rhythm variations, which indicate emotional responses. For instance, pleasant emotions increase heart rate,65 while HRV decreases in response to unpleasant emotions such as fear or sadness.64 The acceleration or deceleration of heart rate is linked to heart rate asymmetry (HRA), a promising marker for diseases affecting neuro-cardiac control.66 Positive heart rate responses lower blood pressure and muscle tension,67 whereas negative emotions raise the risk of coronary heart disease (CHD).68
Thermal imaging
Thermal imaging, particularly with infrared (IR) cameras, is a non-invasive method that measures skin temperature by detecting electromagnetic radiation in the 0.7-14.0 µm wavelength range.6 These cameras, often integrated into robots or smartphones, are widely used to study emotional responses by tracking thermal variations in facial regions, such as the eyes, cheeks, forehead, nasal tip, and mouth.69 Thermal imaging-based studies have grown significantly in the last decade, with over 2,500 articles indexed in PubMed, 970 of which were published in the last 3 years. However, publicly available thermal imaging databases remain scarce. Additionally, thermal profiles can vary significantly between individuals, even under controlled environmental conditions, complicating their interpretation.70,71 More research is needed to explore the potential of thermal imaging in diverse contexts and populations.
Skin conductance
SC, often measured via electrodermal activity (EDA), is a key indicator of emotional arousal.72 Electrodes measure the skin's electrical properties as sweat secretion changes, primarily from eccrine sweat glands, which are activated by emotional stimuli.72 Positive and negative emotions both influence SC responses.73,74 SC has been used in research on stress, anxiety, fear, emotion, and psychopathology, and is also employed in behavioral therapies such as biofeedback to help individuals regulate their emotional responses.2 SC measurements are commonly taken from the fingers, hands, and feet, which are the most responsive areas, while regions like the armpits and back show the poorest responses.72
Bracelet
Wearable devices, such as smart bracelets or wristbands, are increasingly used in affective computing to monitor and interpret emotional and physiological states. These devices often include sensors for heart rate, SC, and movement (accelerometers),75,76 capturing real-time emotional data.77
Emotion recognition using multimodal methods: a decade of research
Multimodal methods integrate various signals to capture a more comprehensive representation of emotions, often leading to higher accuracy.78,79 Efficient means can aid in emotion recognition, since emotion expression occurs throughout the body as a temporal arrangement of multimodal signals.39
In the last 10 years, there has been an increase in studies focusing on emotion recognition using multimodal methods, but these are still much less common. In a PubMed search limited to the period 2010-2023, we obtained 2,167 results for the expression "multimodal emotions." For comparison, our search for monomodal methods (limited to the same period) yielded 7,009 results for "emotions EEG," 5,228 results for "emotions speech," 3,749 results for "emotions skin conductance," 1,369 results for "emotions ECG," and 22 results for "emotions bracelet." Multimodal studies are not covered further in this review.
AFFECTIVE COMPUTING TOOLS FOR DIAGNOSING AND MANAGING EMOTIONAL COMPONENTS OF DISEASES
Emotions have long been left out of medical and scientific research. Now, however, there is mounting evidence that emotions - especially stress, anxiety, anger, and depression - are important factors with serious and significant implications on health.5 A cursory search of the PubMed platform highlights the increasing research interest in the role of emotions in health, with exponential growth in recent years (Figure 3).

Affective computing enables research and understanding of the relationship between human emotions and health. This, in turn, can offer useful insights for medicine. For Picard,5 the goal of new affective technologies for medicine should be to help medical professionals meet patients' health needs - both emotional and non-emotional - in a balanced, respectful, and intelligent way.
Luneski et al.2 published a review of the uses and advances of affective computing and its applications in medicine up to 2010. As an update of this review, we present below the main applications of affective computing in the field of health and the benefits of incorporating emotion recognition systems in each of them.
Methods for emotion detection in healthcare can be non-invasive (such as video detection of body movement and facial expressions, or speech) or low-invasive (such as EEG or ECG). Non- or low-invasive methods are always preferable, and sometimes necessary, such as to support a diagnosis of autism spectrum disorder.80
Among the methods, facial expressions are strongly correlated with emotions and represent a non-invasive approach.37,81-83
Diseases and disorders
Health is related to homeostasis, which is the body's state of balance. When this balance is destabilized, the normal functioning of the body can be affected, promoting the onset of diseases and disorders. Several factors can affect the balance of the body, among them emotions.1 There are many different types of diseases and disorders, each of which has an emotional basis or at the very least some emotional component.3
Affective computing tools can help identify the emotional components associated with diseases and disorders to aid in the prevention, diagnosis, and treatment of these conditions. We reviewed studies that used technologies to prevent, identify, treat, or predict diseases and disorders that have some relationship to affective states. Supplementary Tables S4-S9 list the selected articles.
Alexithymia
Autism spectrum disorder (ASD), or simply autism, is a complex disorder that includes several levels of impairment and may or may not incorporate the trait of alexithymia (difficulty in identifying, interpreting, and describing one's and others' emotions).84-86 Although this restriction is unspecific to ASD, it is associated with some of the disorders on the spectrum, serving as a diagnostic criterion.
ASD has significant heterogeneity and alexithymia, when present, has several levels of severity. One of the criteria for diagnosis of ASD is the reduced use of facial expressions. Besides difficulty recognizing emotions, individuals with alexithymia have decreased facial expression.80,86 Due to this trait, affective computing has the potential to help diagnose ASD.
Furthermore, current intervention techniques for children on the autism spectrum suggest that many of them can make progress in recognizing and expressing emotions when they learn emotions. Therefore, some tools from affective computing have also been used in the emotional education of persons with ASD.
Supplementary Table S4 lists the studies included in this review that used affective computing tools in the diagnosis and/or emotional education of people with alexithymia.
Pain care and the distinction of pain and emotions
Like emotions, pain can be detected by monitoring facial expression, body movement, electromyography,87 SC, pupil dilation, ECG, and other parameters. Detecting facial expressions that differ from emotion-related expression was among the methods best able to identify pain.88,89 However, pain is also associated with specific negative emotions, such as agitation and disgust.88,90 The Facial Action Coding System (FACS)37 also describes facial expressions of pain.
The detection of pain-related emotions may offer possible solutions for the treatment of chronic pain, in which emotional factors define the success of rehabilitation.91 Pain-related emotions are a barrier to recovery from chronic pain, and systems trained to detect emotions automatically may be a potential solution.91
Affective computing tools can be applied in monitoring,92 prevention,93 identification, and management of pain.
Supplementary Table S5 lists the studies included in this review which used affective computing tools in the diagnosis of pain.
Disorders of consciousness (DOC)
Patients with DOC, such as coma and vegetative state, suffer from reduced mobility and are generally unable to adequately express emotions.94 There is a paucity of studies that aim to differentiate emotions via EEG in individuals with these conditions.
Affective computing tools have been investigated as a potential tool to assess and manage emotional and psychological distress in patients with disturbed consciousness, such as those in a vegetative or minimally conscious state.95 These systems can help healthcare professionals track patients' feelings and behaviors over time, which would allow them to detect patterns that may indicate DOCs and could help provide more effective and earlier treatment. Affective computing technologies, such as facial expression recognition96 and physiological detection,97 can also be used to assess the emotional and physiological states of patients already in such states and provide appropriate interventions to improve their well-being. For example, affective computing can be used to design interactive environments or communication systems that can respond to patients' emotional cues and provide appropriate feedback or stimulation.98
Supplementary Table S6 lists the studies included in this review which used affective computing tools in the diagnosis of DOC.
Neurodegenerative diseases
People with neurodegenerative diseases have altered perception and socioemotional response.99 For example, patients with behavioral variant frontotemporal dementia (bvFTD) have a low resting state of emotional arousal and emotional blunting that may be verifiable via SC. The insula and cingulate cortex and the amygdala are involved in SC regulation and are areas affected by bvFTD.100
Studies highlight the relationship between SC and feelings of threat, suggesting it is a predictor of emotional responses to stressful life events.101 Compared to patients with Alzheimer's disease and healthy control subjects, patients with bvFTD - even though they exhibit a startle response - have low levels of SC.100
Emotion recognition could be a diagnostic aid for neurodegenerative diseases in their early stages. However, apathy, mobility difficulties, and changes in emotional regulation associated with normal aging make it difficult to identify emotional disorders when such diseases are already established. To address this challenge, the development of experimental protocols that assess emotional state through home-based affective computing technologies (e.g., smart homes) has been proposed.102
Smith et al.103 state that affective computing can become an important care delivery tool for late-stage affective and cognitive disorders. They can serve to identify late-life cognitive and mood disorders, including depression and Alzheimer's disease. Affective computing technologies can offer objective biomarkers and tools for early detection, monitoring of treatment response, and tracking disease progression, as well as provide a more comprehensive understanding of patients' daily lives. Pontieri et al.104 compared facial emotion recognition performances between patients with progressive supranuclear palsy (PSP) and a large group of patients with Parkinson's disease (PD), examining the different encoding thresholds for emotional faces in PSP and PD.
Supplementary Table S7 lists the studies included in this review which used affective computing tools in the diagnosis of neurodegenerative diseases.
Depression
Prolonged sadness can lead to depression, which in turn is one of the most common mental disorders and is a leading cause of disability worldwide. This disorder affects people of all ages, including older adults. Geriatric depression has become a public health problem. Its effect on health increases when it is comorbid with a chronic medical condition.1
Affective computing can provide tools for the identification of depression, allowing a more objective analysis of its different levels of severity. Such technologies can also serve as a support tool for the treatment of patients.
Supplementary Table S8 lists the studies included in this review which used affective computing tools in the diagnosis and/or treatment of depressive disorder.
Stress
Stress is a state of emotional arousal that promotes the onset of disease.1 Therefore, it is a risk factor for poor health, necessitating better measurements.
Affective computing tools can assist in identifying a patient's stress levels, providing objective data for the patient themselves and health professionals, and can play a crucial role in the diagnosis of stress by integrating emotional recognition technologies and biometric data analysis to monitor physiological and behavioral reactions in real time. Recent studies, such as that by Wang et al.,105 have shown that analysis of facial expressions, vocal tone, and physiological patterns, such as heart rate and HRV, can help identify stress states more accurately and efficiently. These technologies allow for early and personalized interventions, enhancing the effectiveness of stress diagnosis and treatment while providing valuable insights into individuals' mental health.105
Supplementary Table S9 lists the studies included in this review which used affective computing tools in the diagnosis of stress.
Irritable bowel syndrome
IBS is a multifactorial condition which causes visceral hypersensitivity and pain. IBS is related to emotional states of anxiety, depression, and stress reaction and exposure. Such intense emotional conditions lead to an imbalance in the relationships between the ANS and the hypothalamic-pituitary-adrenal axis, which alters the homeostasis of the gut biota, leading to development of IBS.106
There are no studies that demonstrate or propose the use of emotion recognition as a tool to aid in the recognition and/or treatment of IBS; however, emotion monitoring may be useful, as is already being done in bipolar disorder.107,108 As noted by Pellissier & Bonaz,106 IBS is a multifactorial disorder involving biological, psychological, and social stimuli. Monitoring emotional state and recognizing stress, one of the main triggers of gastrointestinal symptoms, may assist in the control of the syndrome.
Affective computing can help detect patterns in patients' symptoms and behaviors, allowing for a more accurate diagnosis of IBS. It can also be used to provide personalized care and treatment plans tailored to the needs of the individual. In addition, it can help to better understand the psychological aspects of IBS, as well as its social and cultural impacts. For example, affective computing could be used to track a patient's emotional responses to stress,109 as well as collect, analyze, and interpret data related to the expression of IBS symptoms, including physiological and emotional information.110 These data could be used to identify patterns and predictors of IBS flares and to inform diagnosis and treatment. Finally, affective technologies could be used to monitor patients' progress and response to treatment.111
Although there are numerous possibilities for the practical application of affective computing in the diagnosis and emotional treatment of people with IBS, we did not find any studies on this topic.
Smart homes and environments in emotional well-being and health care
The number of studies focusing on developing smart homes - monitored environments enriched with non-invasive sensors located in various rooms - and other smart environments has grown. Among the many uses of these spaces, assisted living for older adults is one of the most studied and growing applications. Smart homes had the initial purpose of providing security and energy savings; however, their focus has now turned to health care, wellness, and comfort to assist people with disabilities, older adults, and people with reduced mobility (e.g., wheelchair users). The authors of a critical review on smart environments emphasize that these environments need to provide emotional well-being and, therefore, should provide entertainment and social experiences.112
Another emerging use of smart technology in health care systems is the monitoring and care of patients with psychiatric disorders, such as bipolar disorder. For example, Grünerbl et al.107 proposed using a mobile phone-based system that monitors the behavioral aspects of patients with the condition, while Prociow et al.108 aim to provide an early warning system mediated by wearable and environmental sensors. The signals captured by these sensors offer clues about the patient's emotional state, which may lie between depression, normality, and mania, allowing the psychiatrist to follow up. Monitoring mediated by such systems would make life easier for patients with bipolar disorder and support their treatment by providing vital information to their psychiatrist. Furthermore, this information ensures better decision-making, supporting implementation of any necessary therapeutic measures, and may prevent incidents, including suicide.107,108
Additionally, there is a wide range of work focused on the use of emotional state recognition47,113 for behaviors linked to fatigue,114 stress, drunkenness, and other dangerous behaviors of motor vehicle drivers.115,116 Several studies have also focused on drivers' emotions to prevent automobile accidents and thus promote the health and safety of road users.117
Recognizing emotions in the real world
There are considerable differences between real-world (or "in-the-wild") environments and controlled (laboratory/posed) environments. According to Tzirakis et al, these may include lack of noise or reverberation, unobstructed visibility, poor lighting, and varied/non-uniform backgrounds. Participants in controlled environments are also restricted from moving as freely as they would in the real world. Similarly, posed faces display a limited range of expressions and lack the subtle nuances that naturally occur in realistic, spontaneous environments.118,119 All these issues interfere with the efficiency of emotion recognition methods.118
Applying emotion recognition methods in uncontrolled environments tends to achieve much lower accuracy and effectiveness than in controlled environments.120 However, several studies have achieved significant results.121,122 For example, Liu et al.122 achieved 66.9% accuracy and an F1-score of 40.8% in classifying images extracted from the Aff-Wild2 dataset of Kollias & Zafeiriou,123 which combines the ResNet and Bidirectional Long Short-Term Memory (BLSTM).
There have also been specific studies aimed at circumventing problems brought about by real-world emotion recognition,124 some of which aimed to address particular issues such as recognizing facial expressions with occlusion125-128 and noisy environments.129 Studies based on multimodal systems, which combine face (video), text (written), and speech (acoustic, words) data, are able to deal with the noise and occlusions characteristic of in-the-wild data.130-133
Mollahosseini et al.34 point out the scarcity of in-the-wild datasets with labeled images and survey the principal available datasets. They go on to propose AffectNet, a set of 1 million in-the-wild facial images of which 450,000 have been manually annotated into eight discrete and valence-arousal emotional categories.
RESULTS
Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes
We found a substantial amount of studies which scored low in our evaluation, as shown in Table 1.
| Subjects | Data | Code | Link | Peng's36 metric | Our metric | Reference |
|---|---|---|---|---|---|---|
| 5 | 0 | 0 | 0 | 0 | 0.698 | Wang134 |
| 5 | 0 | 0 | 0 | 0 | 0.698 | Lin135 |
| 6 | 0 | 0 | 0 | 0 | 0.778 | Pan136 |
| 10 | 0 | 0 | 0 | 0 | 1 | Yu137 |
| 10 | 0 | 0 | 0 | 0 | 1 | Li & Lu61 |
| 10 | 0 | 0 | 0 | 0 | 1 | Boccanfuso138 |
| 12 | 0 | 0 | 0 | 0 | 1.079 | Wei139 |
| 14 | 0 | 0 | 0 | 0 | 1.146 | Jerritta140 |
| 14 | 0 | 0 | 0 | 0 | 1.146 | Alsolamy & Fattouh141 |
| 16 | 0 | 0 | 0 | 0 | 1.204 | Petrantonakis & Hadjileontiadis142 |
| 17 | 0 | 0 | 0 | 0 | 1.23 | Goulart143 |
| 19 | 0 | 0 | 0 | 0 | 1.278 | Jalilifard144 |
| 19 | 0 | 0 | 0 | 0 | 1.278 | Shahabi & Moghimi145 |
| 20 | 0 | 0 | 0 | 0 | 1.301 | Zhang146 |
| 20 | 0 | 0 | 0 | 0 | 1.301 | Murugappan147 |
| 20 | 0 | 0 | 0 | 0 | 1.301 | Taran & Bajaj148 |
| 20 | 0 | 0 | 0 | 0 | 1.301 | Khare & Bajaj149 |
| 21 | 0 | 0 | 0 | 0 | 1.322 | Mehmoo150 |
| 21 | 0 | 0 | 0 | 0 | 1.322 | Mehmood & Lee151 |
| 21 | 0 | 0 | 0 | 0 | 1.322 | Raheel152 |
| 25 | 0 | 0 | 0 | 0 | 1.397 | Dissanayake153 |
| 25 | 0 | 0 | 0 | 0 | 1.397 | Guo154 |
| 25 | 0 | 0 | 0 | 0 | 1.397 | Shu75 |
| 25 | 0 | 0 | 0 | 0 | 1.397 | Yu & Tapus155 |
| 26 | 0 | 0 | 0 | 0 | 1.414 | Lin156 |
| 26 | 0 | 0 | 0 | 0 | 1.414 | Basu157 |
| 30 | 0 | 0 | 0 | 0 | 1.477 | Bhatti158 |
| 30 | 0 | 0 | 0 | 0 | 1.477 | Zhuang159 |
| 33 | 0 | 0 | 0 | 0 | 1.518 | Tarvainen160 |
| 35 | 0 | 0 | 0 | 0 | 1.544 | Kolli161 |
| 36 | 0 | 0 | 0 | 0 | 1.556 | Mikhail162 |
| 44 | 0 | 0 | 0 | 0 | 1.643 | Cruz-Albarran163 |
| 50 | 0 | 0 | 0 | 0 | 1.698 | Quiroz164 |
| 59 | 0 | 0 | 0 | 0 | 1.77 | Zhang76 |
| 59 | 0 | 0 | 0 | 0 | 1.77 | Cui165 |
| 60 | 0 | 0 | 0 | 0 | 1.778 | Selvaraj166 |
| 67 | 0 | 0 | 0 | 0 | 1.826 | Nayak167 |
| 75 | 0 | 0 | 0 | 0 | 1.875 | Agrafioti168 |
| 84 | 0 | 0 | 0 | 0 | 1.924 | Xun & Zheng169 |
| 154 | 0 | 0 | 0 | 0 | 2.187 | Wan-Hui170 |
| 25┼ | 1╬ | 0 | 0 | 1 | 2.397 | Xianhai171 |
| 254 | 0 | 0 | 0 | 0 | 2.404 | Wu172 |
| 26 | 1╬ | 0 | 0 | 1 | 2.414 | Ferdinando173 |
| 26 | 1 | 0 | 0 | 1 | 2.414 | Chen174 |
| 26 | 1╬ | 0 | 0 | 1 | 2.414 | Nguyen175 |
| 27 | 1╬ | 0 | 0 | 1 | 2.431 | Ali176 |
| 27 | 1╬ | 0 | 0 | 1 | 2.431 | Wei177 |
| 32 | 1╬ | 0 | 0 | 1 | 2.505 | Vijayan178 |
| 32 | 1╬ | 0 | 0 | 1 | 2.505 | Zamanian & Farsi179 |
| 32 | 1 | 0 | 0 | 1 | 2.505 | Chen180 |
| 391 | 0 | 0 | 0 | 0 | 2.592 | Jing181 |
| 391 | 0 | 0 | 0 | 0 | 2.592 | Xu & Liu182 |
| 77 | 1╬ | 0 | 0 | 1 | 2.886 | Brás183 |
| 975 | 0 | 0 | 0 | 0 | 2.989 | Alghowinem184 |
| 28 | 1 | 1 | 0 | 2 | 3.447 | Goulart185 |
| 32 | 1╬ | 1 | 0 | 2 | 3.505 | Balan186 |
| 2,340 | 1§ | 0 | 0 | 1 | 4.369 | Sarath187 |
The higher the value, the higher the reproducibility of the study and the larger the sample size.
┼ The samples correspond to 25 days instead of participants.
╬ Public data.
§ The data refers to an image database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible camera and an infrared (IR) thermal camera (TC), with lighting provided from three different directions.
Considering only reproducibility scores, which range from 0 to 3, most articles scored between 0 and 1. Only two studies achieved a score of 2, as they provided both the data and codes used. However, most studies with available data were due to the use of already public data.
Taking into account only the number of participants, the population sample of the articles was not significant. Most of the monomodal studies analyzed (34.48%) had samples of fewer than 30 individuals and only 12.06% had more than 100 participants; and none had a population greater than 1,000 participants. Of the reviewed studies, 72.41% performed their analyses on a population of fewer than 5 participants.
Considering both reproducibility and sample size, we obtained our metric (equation 1), whose distribution is shown in Figure 4. The figure highlights the low quality of the articles overall, implying low reliability of the results. Only one included study obtained a score above 4, and two obtained a score above 3.

Sarath et al.187 had the best result, with a score of 4.369. This study sought to examine the emotions hidden in the face, including stress and anxiety, through facial temperature measurement.187 Balan et al.186 obtained a score of 3.505. The goal of their study was to define accurate methods for classifying the level of fear. In future research, the authors intend to develop a phobia treatment system based on gradual exposure to virtual reality, which automatically determines fear EEG levels and biophysical data.186 Goulart et al.185 achieved a score of 3.447. The purpose of their research was to propose an experimental design for analysis five emotions (disgust, fear, joy, sadness, and surprise) in children aged 7 to 11 years, by means of changes in facial emissivity detected by infrared thermal imaging (IRTI).
Two among the three highest-scoring articles in our review established relationships between emotions and health, strengthening the argument for the applicability of affective computational tools in the health sciences. Sarath et al.187 discuss the use of a computational tool that identifies emotions, among them stress and anxiety, two factors that can increase the risks of sudden cardiac arrest, a leading cause of mortality and morbidity in affluent societies.188 Balan et al.186 analyzed fear levels with the future goal of creating a tool for the treatment of phobia.
Finally, based on the accuracy of the classification of emotions in the studies surveyed, we checked which signal was most effective in classifying each of Ekman's six basic emotions. Supplementary Figure S2 shows the average percentage across studies for each signal used.
We observed that some signals served the purpose of identifying emotions better than other methods. In addition, we identified that specific signals recognize certain emotions more efficiently than others. We were thus able to associate each emotion with a capture method or physiological signal that is most efficient at identifying it.
Out of the 109 articles initially analyzed, 64 were monomodal, employing EEG (n=26), ECG (n=17), TC (n=11), accelerometry (n=6), and GSR/SC (n=4). The results showed that TC had the best average accuracy for detection of happiness (83.48%), while ECG was the best for detection of sad (79.83% average accuracy). Finally, smart bracelets had the best average accuracy for identifying fear, anger, disgust, and surprise, at 82.57, 89.30, 84.91, and 88.01%, respectively. These results show that the three best monomodal tools for studying basic emotions are bracelets or wristbands, thermal imaging, and ECG.
One caveat is warranted: accuracy is not the best statistical metric for model validation, as will be further addressed in the discussion. Therefore, these results are still only indicative and require further studies for actual validation.
The impact of affective computing in health: gaps and insights from current research
There is a limited amount of research on the application of affective computing tools to health. While some studies have linked these areas, little research has focused on capturing and identifying emotions and applying this to the prevention and treatment of diseases. Much research had used affective technologies to predict disease, propose treatments, or teach patients to recognize and manage emotions without, however, detecting these emotions. In the case of alexithymia, for example, some research focuses on emotional education. As for pain, depression, and stress, the disorder or condition itself may, in some studies, become the object of emotional capture and identification.
Supplementary Table S10 presents the results of our search. The largest number of studies retrieved concerned neurodegenerative diseases. We believe this was due to the lack of a more targeted filter. The filters used allowed for a large volume of articles and reviews on the topic to be retrieved by the Google Scholar search tool. Few of these searches applied affective computing tools to neurodegenerative diseases. One of the articles31 appeared in the results for disorder of consciousness. Since the focus of the study was PD, we decided to keep it among the results for neurodegenerative diseases.
IBS showed only one result in Google Scholar and none in PubMed. We chose to keep this disease in our discussion because we realized, in our rationale, that there is a potential research field for affective computing tools to aid in treatment of this syndrome.
There were fewer results for depression and stress. We believe this was due to the use of filter 3, in which we added the word emotion to the search query. This was done because the words stress and depression, by themselves, result in retrieval of publications broadly related to affectivity, not specifically emotions. Targeting allowed for targeted results focusing on emotions. This term was not added to the other search strings because, conversely, not targeting emotions provided results related to the field of affectivity in general.
In our final sample, most studies focused on the recognition of depressive disorders. Wang et al.189 used sensors to assess gait. Dham et al.29 used multimodal techniques to recognize depression from a combination of audio, visual, and text analysis. Liu et al.,28 He & Cao,30 and Hansen et al.33 used classification and measurement tools and facial recognition to identify depression through speech. Guo H. et al.32 reported a method for recognizing depression through facial expressions. Huang et al.190 compared several machine learning methods to identify a model that most accurately predicted suicidal ideation and depression level in school-aged adolescents.
After reviewing the articles, we applied our quality metric (Table 2) to identify replicability and methodological reliability.
| Subjects | Data | Code | Link | Peng's36 | Our metric | Reference |
|---|---|---|---|---|---|---|
| 8 | 0 | 0 | 0 | 0 | 0.903 | He191 |
| 10 | 0 | 0 | 0 | 0 | 1.000 | Liu & Jiang192 |
| 11 | 0 | 0 | 0 | 0 | 1.041 | Varotto193 |
| 12┼ | 1╬ | 1 | 0 | 2 | 3.079 | Perez-Toro194 |
| 14 | 0 | 0 | 0 | 0 | 1.146 | Pan195 |
| 15 | 0 | 0 | 0 | 0 | 1.176 | Hosseini196 |
| 15 | 0 | 0 | 0 | 0 | 1.176 | Moon & Qu197 |
| 18 | 0 | 0 | 0 | 0 | 1.255 | Huang94 |
| 18 | 0 | 0 | 0 | 0 | 1.255 | Zolyomi & Snyder198 |
| 20 | 1 | 1 | 1 | 3 | 4.301 | Prossinger199 |
| 20 | 0 | 0 | 0 | 0 | 1.301 | Bourvis200 |
| 22 | 0 | 0 | 0 | 0 | 1.342 | ArulDass & Jayagopal201 |
| 24 | 1╬ | 0 | 0 | 1 | 2.380 | Dham29 |
| 30 | 0 | 0 | 0 | 0 | 1.477 | Akbulut202 |
| 42 | 0 | 1 | 0 | 1 | 2.623 | Hansen33 |
| 50 | 1 | 0 | 0 | 1 | 2.699 | Sparrow203 |
| 52 | 0 | 0 | 0 | 0 | 1.716 | Filippou204 |
| 55 | 0 | 0 | 0 | 0 | 1.740 | Moretta205 |
| 65 | 0 | 0 | 0 | 0 | 1.813 | Goldstein27 |
| 74 | 1 | 1 | 1 | 3 | 4.869 | Atee206 |
| 77 | 1 | 1 | 1 | 3 | 4.886 | Kang207 |
| 83 | 1 | 0 | 0 | 1 | 2.919 | Garcia-Cordero208 |
| 85 | 0 | 0 | 0 | 0 | 1.929 | Wu209 |
| 85 | 1 | 0 | 0 | 1 | 2.929 | Campbell210 |
| 95 | 0 | 0 | 0 | 0 | 1.978 | Wang211 |
| 104 | 1 | 0 | 0 | 1 | 3.017 | Guo32 |
| 109 | 1 | 1 | 1 | 3 | 5.037 | Birba212 |
| 184 | 0 | 0 | 0 | 0 | 2.265 | Liu28 |
| 195 | 1 | 0 | 0 | 1 | 3.290 | Pontieri104 |
| 300 | 1╬ | 0 | 0 | 1 | 3.477 | He & Cao30 |
| 873§ | 1 | 1 | 0 | 2 | 4.941 | Caldo213 |
| 10,234 | 0 | 0 | 0 | 0 | 4.010 | Huang190 |
| 131,025|| | 1╬ | 0 | 0 | 1 | 6.117 | Dar31 |
The higher the value, the higher the reproducibility of the study and the larger the sample size.
┼ The samples correspond to 12 hours of audio instead of participants.
╬ Public data.
§ The samples correspond to 873 URLs instead of participants.
|| The samples correspond to 131,025 seconds instead of participants.
We highlight the seven articles with scores above 4, out of a total of 34. First among them was Dar et al.,31 with a score of 6.11. The researchers applied the BCI emotion recognition system to patients with OCD. This study was cited in nine other articles,214-222 as a reference on the use of affective computing tools in health care. Its results show that the proposed BCI system can be a promising tool for emotion recognition in patients with DOC.
In second place, Birba et al.,212 with a score of 5.03, used a multimodal combination of behavioral, neuroanatomical (voxel-based morphometry [VBM]), and resting-state functional connectivity (rsFC) (fMRI and high-density EEG derivatives [hd-EEG]) measures to examine key signatures of naturalistic understanding of social actions and speech in patients with PD and bvFTD, compared to healthy controls and patients with Alzheimer's disease.
In the third position was Caldo et al.213 with a score of 4.94. In this study, the researchers developed a machine-learning algorithm with the goal of distinguishing discrete emotional fingerprints from web pages related to back pain.
Kang et al.207 came fourth, with a score of 4.88. Their paper presented the K-EmoPhone, a set of multimodal and physiological data collected from participants' smartphones and self-reported affect states, including emotions, stress, attention, and task interruption, acquired by the experimental sampling method.
In fifth place, with a score of 4.86, Atee et al.206 described a new method, called PainChek, aimed at assessing and monitoring pain, as well as profiling patients and synchronizing data.
Sixth was Prossinger et al.,199 with a score of 4.30. In this study, the researchers use AI to analyze facial expressions and determine the distances between the affective states Neutral-Pain, Neutral-Pleasure, and Pleasure-Pain.
Finally, in seventh place with a score of 4.01, Huang et al.190 proposed a method of emotional diagnosis through the recognition of facial expressions.
The two highest scores were achieved by studies on neurodegenerative diseases. The highest number of studies with scores above 4 was research that related affective computing to pain. Of the seven articles ranked highest, three were about pain. Depression and stress had one article each with a score above 4.
Most of the included studies (44.11%) scored less than 2.0 points, including all studies on DOC and alexithymia. Of the seven research papers on depression, five had scores between 1.0 and 2.0. Only one article on depression and two on neurodegenerative diseases were between these minimum limits.
DISCUSSION AND CHALLENGES
We begin by presenting a discussion on affective computing research, based on our analysis. Next, we evaluate studies on the application of affective technologies to the prediction, diagnosis, and treatment of various diseases. Finally, we highlight the challenges faced by the field, as identified in our investigations.
Technologies employed for emotion recognition: shortcomings
The recognition of emotions is routinely carried out in our lives and facilitates interpersonal interaction, physiological reactions of emotional origin, and understanding of mental states. The homeostasis of the human body is influenced by physiological responses to the emotional context, and its imbalance can trigger diseases. Many studies reveal that emotions such as anger, anxiety, depression, and stress are significant medical factors.5
In recent years, an exponential growth of 233.54% in the study of emotions and their relationship to health and medicine has occurred (Figure 3). This growth reveals the scientific interest in the topic as stated by Consedine & Moskowitz.223
However, during our literature survey, we were faced with a systematic lack of concrete health applications of emotion detection techniques. For example, few articles used the term "affective medicine" explicitly. Despite the scarcity of studies as focus on the application of computational tools applied to affective medicine (described in the Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes section), there is a great potential and even a need to develop methods to capture human emotions and apply this to health. Affective computing emerges to provide technological means that capture and identify emotions and thus help professionals provide emotional support in the treatment of an illness.5
Despite the significant growth of studies exploring the relationship between emotions and health, as well as the promising potential of this field, several critical deficiencies persist in the study of emotions, their detection, and their application in health contexts. This review has identified that current technologies developed for emotion detection remain inadequate for practical implementation in human health.
In this work, we systematically identified gaps and evaluated existing studies by employing an evaluative scale inspired by the framework proposed by Peng.36,41 To address these challenges, we developed a metric grounded in Peng's methodology, which is widely recognized in the literature. While not without limitations, this metric provides an objective tool for assessing the methodological quality of the reviewed studies, surpassing the constraints of exclusively qualitative analyses.
Our findings reveal that many studies fail to meet essential criteria for reproducibility and scientific rigor. Common shortcomings include insufficient sample sizes and the lack of consistently validated methods. These limitations underscore the need for more robust methodological frameworks to ensure reliable and impactful advancements in the intersection of emotional recognition and health research.
Overfitting and underfitting are phenomena that occur during the training of neural networks. Overfitting occurs when a statistical model effectively fits the previously observed data set but fails to accurately predict new results. One of the causes of this phenomenon is sample size. When the training set is too small or has less representative or very noisy data, the results are compromised. Larger samples more accurately represent the value of the population, while smaller samples can present a statistically non-significant result. More heterogeneous studies require a large sample size to obtain accurate results.224 On the other hand, underfitting occurs when the model is not complicated enough and focuses too little on the training data.225 The problem with underfitting is that training is insufficient and learning accuracy is low.226 As a result, the model can neither fit the training set nor generalize well to new data.
In our analysis, we included the sample size criterion to identify one of the problems with validating research into affective computing. The majority of studies in the area present scarce data, compromising their results with overfitting. In turn, we excluded gender and age to avoid underfitting our analysis. Both influence the generalization of the data and could compromise our results, since some studies did not provide the age of the participants, others focused on very different age groups and still others on a single gender.
Among the monomodal articles analyzed, most have methodological inconsistencies or gaps, which prevent analysis of the reported tools. The lack of access to data and/or the size of the population jeopardize the quality of the results.
Additionally, studies - as a consensus - use accuracy as the metric to validate the trained classificatory methods for the various machine learning models used. However, the accuracy metric is insufficient to validate methodological reliability because it is subject to biases, such as when there is an imbalance between classes; in fact, it is one of the worst possible metrics for validation purposes. Among the metrics adequate for such a task would be the F1-score and the combined analysis of metrics such as specificity, accuracy, precision, and sensitivity, which were mostly uncalculated.
In relation to gaps highlighted by the proposed metrics, evidence of the low reliability of the results highlights the need for more general studies before their implementation in the health field, and justify why emotion recognition methods are not currently applied to health. What would explain this low study quality remains to be determined.
Affective computing technologies in healthcare: focus on neurodegenerative diseases and emotional diagnostics
As raised in the Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes section, several potential uses of affective computing in healthcare can be explored. We highlight the development of methods for monitoring and tracking patients, as has already been applied to individuals with psychiatric conditions such as bipolar disorder,107,108 allowing medical specialists or psychiatrists to follow patients' progress and avoid incidents, technologies that can be improved and applied to other disorders such as schizophrenia. It also has great potential in helping patients with DOC94 who are otherwise partially or completely unable to communicate.
Despite the potential benefits, we find that, to date, little research had applied technologies to identify emotions in the context of health. Many studies used devices and tools such as EEG, ECG, etc., without focusing on emotion detection. Comparatively, the amount of research relating neurodegenerative diseases to affective computing, with the use of computational tools to analyze the emotional state of people with Alzheimer's disease, Parkinson's disease, and ALS, reveal the importance of this area of study for health. On the other hand, the lack of practical studies on the benefits of affective technologies in the diagnosis and/or treatment of emotional states of people with IBS demonstrates the need to invest in more research focused on affective computing applied to health. We also highlight studies on the detection of depression through voice. According to our surveys, most affective computing research focused on depression uses voice capture tools to diagnose the disorder.
Our analysis of methodological quality of research on affective computing applied to health showed that studies on neurodegenerative diseases performed best. We highlight the work of Dar et al.,31 which has become a reference in the application of affective computing tools in the care of patients with neurodegenerative diseases. This is a particularly promising area, as affective technologies can aid in the recognition and emotional assessment of people with PD, ALS, and dementia.
Research on the application of computational tools in the diagnosis of pain and related emotions also shows promise. Studies have shown good results. This area of research can lead to many advances in the diagnosis and treatment of pain.
There is a lack of qualitative research on the applicability of affective computing in the emotional diagnosis of people with disorders of consciousness. We recognize that this is a difficult area, but it is also an important field in which more studies are sorely needed to ascertain whether affective technologies can be used in the emotional assessment of people with DOC.
Having verified the importance of the applicability of affective computing to health, we highlight the need for more accurate studies. We suggest the use of a text mining tool to obtain more accurate results, with the classification of the most relevant and most necessary themes.
Overcoming challenges in emotion recognition for medical and real-world implementation
Based on the findings of our literature review, we established a methodological roadmap in the format of challenges that need to be overcome (Figure 5). These challenges represent the deficiencies observed in the process of recognizing emotional states which must be solved before adequate, reliable tools can be obtained for medical and general use.

The first challenge is to obtain a gold standard. The complexity related to the concept of emotion itself is an obstacle to a precise definition. There is also great difficulty in establishing a gold standard in interpreting emotions, given the impossibility of directly measuring an individual's emotional state.16,80 Consequently, there is a lack of standardization across studies, as reported by Mauss & Robinson.16
The affective computing literature is beset by a broad lack of a solid foundation. Striking results are validated by subjective elicitation methods. Several studies have questioned the reliability of using self-report questionnaires to assess felt emotions, depositing undue trust in emotion elicitation methods which may partially assure the triggering of the expected emotion.
This aspect warrants particularly careful analysis, because confidence in study results is essential to establishing standards and verifying research integrity. Regardless of its cost, invasiveness, or difficulty, this standard should allow identification of the emotion being felt within a current classification, usually a discrete model.
It can therefore be said that the biggest challenge in the field of emotions is to establish gold standards, whether for emotional models, elicitation methods, detection standards, or interpretation of results. A gold standard is essential for the standardization and reliability of analyses, both to make progress and to ensure safety in the use of new technologies, as well as to ensure future uptake and use by health care providers. Such standards will facilitate the validation and comparison of proposed methods and tools and will allow the construction of curated databases for different signals.
The establishment of a gold standard in affective computing necessitates the implementation of cross-validation with biometric methods. Integrating technologies such as EEG and fMRI is a promising approach for validating emotional recognition algorithms, offering a more precise and robust validation process. These methods provide a deeper understanding of the neural mechanisms underlying emotional responses, facilitating the development of more accurate models for emotion recognition.227 Moreover, incorporating physiological measures, including heart rate and GSR, provides objective references that enhance the reliability of emotional analyses by enabling data triangulation between physiological signals and computational models. This multimodal approach serves to strengthen the validity of the emotional data captured, ensuring a more comprehensive understanding of affective states.
Equally crucial for the establishment of a gold standard is the development of collaborative benchmarks and clear evaluation metrics. The creation of widely accepted benchmarks requires the concerted efforts of interdisciplinary teams, including experts from computer science, psychology, and other related fields. This collaborative approach is vital for ensuring that benchmarks reflect a broad range of real-world applications. Noteworthy initiatives, such as the Emotion Recognition in the Wild project, aim to bridge the gap between technology and real-world scenarios by providing explicit evaluation metrics, including precision, recall, and rates of false positives and false negatives. These metrics are indispensable for validating and refining emotional recognition systems, thereby enhancing their practical applicability in diverse contexts.227
The second challenge is related to elicitation methods. The reliability of emotional stimuli as a method of validation encounters a barrier in cultural diversity. Psychological studies have shown that the primary expressions of emotions are not culturally universal,228 confirming a cultural influence on emotional responses. Emotional expressions tend to be very different between Western and Eastern populations.229 However, automatic facial expression recognition systems disregard these differences, analyzing and recognizing them universally in different cultures.228 Most elicitation databases focus on Western people with different characteristics than Eastern people. Larger and more diverse emotion databases are needed to overcome this.229 Additionally, as noted in Supplementary Table S11, constructed datasets generally use self-validation methods (as listed in Supplementary Table S12) to validate the classified emotions, i.e., the labels of each emotion class. The fact that they are based on self-reported methods places the reliability of the data into question.
The third challenge in validating emotional recognition systems lies in obtaining a comprehensive diversity of signals, including physiological, behavioral, and facial expressions. These signals must be less invasive, cost-effective, and practical to collect, ensuring greater applicability in real-world and clinical contexts. While maintaining control over the research environment is crucial for ensuring data quality and consistency, there is an increasing need to broaden the representativeness of the collected data.
Striking a balance between rigorous control and representativeness is fundamental. Excessively granular data may introduce noise and reduce precision, whereas overly restrictive datasets risk bias and limit the generalizability of developed models. To address this challenge, we propose aligning data collection protocols with established validation standards in AI. These standards advocate for controlled, robust methodologies while promoting the diversity necessary to capture the complexity of human interactions.
Once multiple signals have been collected and classified with reliability equivalent to the gold standard, it becomes possible to explore which signal (or combination of signals) is most effective for emotion detection. This decision should carefully balance reliability, cost, and invasiveness. The answer will inevitably depend on the application context and specific circumstances in which the method is employed, highlighting the importance of adaptive and dynamic solutions.
The fourth challenge is in capturing and combining these signals. A particular combination of signals (multimodal methods) allows more efficiency in recognizing emotions. Parallel to evaluating the signals obtained, one must consider which computational methods are more efficient in performing this automated classification. Several machine learning methods can be employed to obtain reliable tools, with the use of reliable data.
The fifth challenge is related to datasets. We surveyed the main datasets (Supplementary Table S11) and found that emotions are induced effectively in the laboratory,230 which makes it feasible to use emotional datasets generated in laboratory environments. However, lack of generalizability is a problem in the datasets found in the literature. The fact that these data originate from laboratory settings, where images of individuals are captured in the same position, with standardized backgrounds and no visual obstructions, is not the main issue, but rather the homogeneity of the participants, who often belong to similar age groups, ethnicities, and nationalities. This limitation in participant diversity restricts the applicability of the results to a broader and more heterogeneous population. Furthermore, we emphasize the need for a more robust dataset for quantitative validation of study outcomes. Proper sample size calculation is crucial in quantitative research, as it directly influences the statistical validity and reliability of the inferences drawn from the data. This calculation takes into account factors such as margin of error, confidence level, and expected population variability. For studies aiming to represent broad and heterogeneous contexts, sample sizes between 400 and 1,000 individuals are often recommended, assuming a 5% margin of error and a 95% confidence level, as suggested by Krejcie.231 However, the studies reviewed in this analysis had significantly smaller sample sizes, ranging from eight to 300 participants. This reduced sample size may limit the generalizability of results and undermine the identification of more robust patterns, particularly in areas as complex and multifaceted as emotional and behavioral health.24
The sixth challenge is the application of machine learning in emotional recognition for health. Despite significant progress, this technology still faces barriers in emotional classification that need to be overcome.232 As well raised by Luneski et al.,2 affective medicine, linked to affective computing, has gaps in its applications waiting to be filled. These include studies focused on the prevention of emotion-related diseases. We found that emotion recognition technologies have made inroads in healthcare, especially in helping patients with neurological and neurodegenerative diseases such as dementia, bipolar disorder, autism, and disorders of consciousness. However, it is still far from being widely and efficiently used to aid diagnosis, prevention, or maintenance of well-being.
There has been little research into developing personal health systems incorporated into home-based health care systems, as corroborated in the study by Lee & Kim.112 The authors state that research and implementation of monitoring systems that consider the emotional well-being of older adults in smart homes are still scarce. We did find research focusing on late-life depression, but there remains a need to ascertain whether affective technologies can be used to improve the quality of life of older adults.
Finally, the seventh challenge is the difficulty of capturing and identifying signals in the real-world setting. In the real world, data acquisition is subject to background variation, occlusion, and movement (for images); noise, a greater variety of positions, angles, movements, and facial expressions and gestures; and real-time implementation issues, among other adversities.118 We can only achieve an effective, applicable method in the real world when these challenges are overcome.
The main barrier to emotion recognition in real-world environments stems from the lack of control over the factors that make up such an environment. As a result, most studies focus on creating and implementing emotion classification methods in highly controlled (laboratory) environments, which achieve considerable effectiveness, but cannot be directly extrapolated to real-world settings.
Few such studies have real-world health applications; most have focused on developing algorithms. There has been no validation or application to diagnosis, patient monitoring, etc. A scarcity of in-the-wild datasets with labeled images,34 necessary for the development and validation of recognition systems, is also noted.
Calibration for the real world is essential for implementation of affective technologies across diverse health care settings, such as home care, hospital practice, and the like. The prospects for applications of emotional state identification are broad for the near future, but the importance of further studies in real-world applications cannot be overstated.
To overcome these and other challenges, we highlight the importance of additional studies focusing on human emotions seen through the prism of affective computing and affective medicine. There is a pressing need for development of emotion databases that are more inclusive: comprehensive in terms of ethnic and age groups, including a large volume of participants, and, in particular, collecting real-world, heterogeneous data, labeled to allow future analysis using parameters with better-balancing refinements, such as the F1-score method. Note, also, that it is necessary to start by establishing a gold standard so that the next steps can then be carried out.
The challenges detected highlight and justify the dearth of literature. There is no solid evidence base on which to build further studies safely and reliably, including for health applications.
CONCLUSION AND PERSPECTIVES
The two-way relationship between emotion and health is evident and warrants greater attention in scientific research. The use of technology for emotion recognition can serve health in different ways, including aiding in the diagnosis and treatment of diseases and monitoring of patients.
Affective computing tools - i.e., those using the machine's ability to capture, analyze, and identify emotions - have the potential to aid in prevention and health care. However, there are great challenges to the real implementation of affective technology to support health and affective medicine.
In this review, we highlighted the most neglected methods for capturing heartbeats, brain signals, and skin conductivity. It is impossible to qualify the results and identify the best tools for emotional identification due to the methodological inconsistencies identified in the studies involving affective computing technologies. In addition to these inconsistencies -non-reproducibility and low number of participants - we were able to list seven technical challenges that are the main causes for the unfeasibility of these methods for emotion recognition in general, as well as specifically in health.
Among the challenges are the lack of a gold standard and poorly curated classes that make it difficult to reliably implement AI methods, which are based on pattern recognition. Consequently, development of systems applicable to real life, such as smart homes, robots, clinics, etc., is hampered.
We present three broad suggestions that can help overcome these challenges. First, conduct studies focused on obtaining a gold standard. From a well-established gold standard, it is possible to move forward and overcome the challenges presented in Figure 5. Second, conduct studies with larger sample sizes, greater diversity in age and ethnicity of participants, and in less controlled environments, using replicable methodologies and making data and methods available, as established by Peng,41 thus avoiding bias, overfitting, and promoting the generalization of models with greater reliability. Finally, we suggest further exploration of the potential use of emotion detection in healthcare. As mentioned, several potential applications with low cost and invasiveness can promote improvements mainly in prevention and patient follow-up.
SUPPLEMENTARY MATERIALS
Supplementary MaterialACKNOWLEDGEMENTS
We thank Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for supporting the Programa de Bioinformática, Fundação da Universidade Federal do Paraná (FUNPAR), Fundação Araucária - Novo Arranjo de Pesquisa e Inovação (NAPI) Bioinformática (grant PDI-66/2021), Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq; grant 440412/2022-6), and Universidade Federal do Paraná (UFPR) for funding the equipment.








PDF
Print
Send this article by email
How to cite this article
Submit a comment
Mendeley
Pocket




