REVIEW ARTICLE


Open Access Peer-Reviewed

5602

Views

Emotional recognition technologies applied to health: review and challenges

Elsa S. Lima1,*, Camila P. Perico2,*, Bruno T.L. Nichio2, Guilherme T. Linhares1, Amora Schwanka3, Reginaldo D. da Silveira3, Dieval Guizelini2, Luiz A.P. Neves3, Roberto T. Raittz2, Jeroniza N. Marchaukoski2

DOI: 10.47626/1516-4446-2024-3963

Epub: April 22 2025

Braz J Psychiatry. 2025;47:e20243963

 

ABSTRACT:

OBJECTIVE: Emotions affect health and health affects emotions. When properly recognized and interpreted, emotions can aid in the prevention, diagnosis, and treatment of many diseases. Affective computing, the use of computer technology to detect one or more signals associated with human emotions, is a promising field. However, research into the use of emotion recognition in healthcare remains scarce. It is crucial to explore the reasons behind this phenomenon and identify neglected methods with potential for human health. We review methods and technologies used in emotion recognition and their applications in healthcare, highlighting methods not discussed in previous reviews, including electroencephalography and electrocardiography, thermal imaging, bracelets, skin conductance, and audio.
METHODS: A metric based on reproducibility and population was established to assess the quality of included articles. Based on the metrics established, we surveyed and analyzed studies in which affective computing tools were applied to health, to qualify and identify the challenges of the area.
RESULTS: We found many challenges to be overcome in detecting and recognizing human emotions, related to sample size, low study quality, and reproducibility issues. We list and discuss the main current challenges, ways to overcome them, and perspectives for the future, focusing on the application of affective technologies in healthcare and the establishment of a gold standard.
CONCLUSION: Three suggestions are proposed: 1) to conduct studies focused on obtaining a gold standard; 2) to conduct studies with larger sample sizes, greater diversity, and in less controlled environments, using replicable methodologies and making data and methods available; and 3) to further explore the potential use of emotion detection in healthcare.

Keywords:
Emotion; sensors; technology; health; recognition.

FIGURES

Note: How to cite this article: Lima ES, Perico CP, Nichio BTL, Linhares GT, Schwanka A, Silveira RD, et al. Emotional recognition technologies applied to health: review and challenges. Braz J Psychiatry. 2025;47:e20243963. http://doi.org/10.47626/1516-4446-2024-3963
Received: April 19 2021; Accepted: August 16 2021
 

INTRODUCTION

Emotion and health are a two-way street.1 Human emotion, mediated by neurotransmitters and other molecules involved in the physiological and neural triggering of emotions, affects the body's homeostasis. In the same way, imbalances in homeostasis trigger emotions. Loss of homeostasis is strongly linked to the development of illnesses, whether mental (such as depression and anxiety) or physiological (such as irritable bowel syndrome [IBS]). Lima et al.1 provides more details on the relationship between emotions and health as a two-way street, with a focus on older adults and those affected by coronavirus disease 2019 (COVID-19).

Since ancient Greece, philosophers and scientists such as Socrates and Hippocrates have considered emotions as determinants of human health and disease.2 In the 20th century, from the 1970s onwards, biochemistry presented evidence of the relationship between emotions and health. One such discovery was the interaction between molecules and opioid receptors throughout the endocrine, nervous, and digestive systems. This interaction produces physiological effects in the body that affect mood.3 Further biochemical evidence was provided by the discovery of communication between the hypothalamus-pituitary and the immune system, linking emotions to disease.4

Over the past 20 years, the growth of research on the relationship between emotions and health has led to the emergence of the term affective computing, coined by Picard.5 Affective computing is an incipient field that involves the development of algorithms designed to detect one or more signals associated with human emotions. Due to its recent emergence, there are still few review studies available. Among the existing methods, the most affordable and least invasive are the least explored, despite their greater practical applicability and coherence, as they do not cause changes in emotional states due to their low invasiveness.

Within this context, we conducted a review focusing on affective computing and current technologies for the detection and recognition of human emotions as applied to human health. Facing a wide diversity of emotion detection methods and studies - without necessarily having a link to health - we first aim to establish whether current methods are suitable to be applied to human health. We then determine which obstacles and challenges exist that hamper these advances. We focus on the emotion recognition methods less explored in previous studies, such as those of Lunesk et al.2 and Dzedzickis et al.,6 since these are neglected methods in the literature.

In the Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes section, we present a survey of existing applications of emotion recognition in human health and potential uses, grouped by an application regardless of the technology or combination of methods employed. In the Affective computing tools: monomodal and multimodal methods for emotion detection section, we review technologies - initially, by individual signal type (Monomodal studies in emotion detection: tools and techniques for physiological signal analysis), then the combination of signals (Emotion recognition using multimodal methods: a decade of research), and finally the challenges and solutions of real-world implementation (Recognizing emotions in the real world). Finally, in the Discussion and challenges section, we present our considerations and the challenges identified.

 

BACKGROUND

Emotions are a category of expressions associated with a biological and/or neural basis. An emotion is a response to an internal or external stimulus. This stimulus is signaled to the brain, which causes behavioral and physiological reactions.7 Faced with an emotional stimulus, the brain releases substances that generate physiological responses that affect the functional balance of the organism.8

Studies in the field of affective neuroscience have revealed an interactive signaling network whereby the peripheral nervous system and the central nervous system receive and generate the experience of emotion.9 This experience activates a brain-body connection10 through substances (neurotransmitters, steroids, and peptides) called emotion molecules,3 which stimulate sympathetic and parasympathetic pathways, shaping physiological activity to respond to constant changes in the internal and external environment.11 The result of this variation translates to homeostatic alterations at the moment of an emotional reaction.

Devising algorithms to detect the various signals potentially associated with different human emotions is a complex task. Emotions can be captured by different signals, including temperature, voice variation, electrical signals, etc., and this may be done by measuring one (monomodal) or more (multimodal) such signals.

The concept of affective computing was developed by Picard.12 This line of research has emerged to provide reliable bases for studies that link emotions and computers, through the use of theoretical descriptions of human affective states, as well as their influence on social, cognitive, physical, or other levels of human actions.2 It focuses on sensors and algorithms whose aim is to recognize and communicate emotional responses.13 Detecting, processing, and interpreting emotional responses by means of emotional expression has various medical purposes,2 and constitutes an objective, safe, and efficient way of using emotional information to support well-being and quality of life.14

Technological methods for classifying emotions require models to perform the categorization. Many papers have sought to provide an overview of emotion categorization models.15-20 There are three general approaches to modeling and classifying emotions: discrete, dimensional, and evaluative. The most widely used and accepted in the literature are the discrete, particularly Ekman's model of six basic emotions, and the dimensional, especially the arousal-valence model.

The discrete model has its origins in studies focused on facial expressions21 and is linked to evolutionary stimuli and triggers. Studies have already linked discrete emotions, called basic emotions, to neural structures.22 Researchers using this theory focus on identifying the biological and/or neural basis of emotions.19 The most widely used sets of basic emotions in the literature correspond to the six basic emotions known as "Ekman's big six": anger, surprise, disgust, enjoyment, fear, and sadness.21 This list is based on studies that involved a survey of facial expressions and present evidence for the existence of universal expressions that are cross-cultural, evidencing that these emotions are "innate" rather than culturally acquired or learned.

In the dimensional model, emotion is understood as a cultural process, learned and expressed differently in each culture. Researchers in this line are more interested in how feelings diverge culturally and how their interpretations vary from subject to subject in different situations. In this theory, infinite affective states exist in a multidimensional space.19 The valence-arousal15 and the approach-avoidance axes are among the leading dimensional models.

As noted above, emotional states can be interpreted monomodally, on the basis of only one signal, or multimodally, combining several parameters.2,12-14,23,24 Affective computing tools capture these signals and classify emotions25 by applying machine learning techniques26 such as classification, regression, unsupervised algorithms, and deep learning. Classification models are trained to assign an emotional category (such as happiness, sadness, anger, among others) to input data. Regression can be used to predict continuous scores related to emotion intensity. Unsupervised algorithms, such as clustering, can be employed to automatically group data into emotional categories without prior labels. Recurrent neural networks can be applied to temporal sequences, such as sentiment analysis of speech or text over time.27 -30 In particular, deep learning involving the use of deep neural networks to automatically learn high-level representations from raw data can automatically extract features from data, eliminating the need for manual extraction.31

Multimodal approaches combine information from different sources to enhance accuracy and have been widely utilized in artificial intelligence (AI) approaches.14 They usually employ analysis of audio signals, such as prosody in speech (pitch, rhythm, intonation), and natural language processing (NLP) techniques for processing written text and extracting relevant information.32,33 Model performance is evaluated using metrics such as accuracy, F1-score, and classification correlation, depending on the specific problem. In this context, specific datasets exist for emotion identification, such as the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) for audio and AffectNet for facial images.34,35

 

METHODS

The objective of this review is to conduct a comprehensive analysis of affective computing tools applied in health care. Initially, we conducted a thorough investigation of studies focusing on technologies for emotion recognition. Subsequently, we systematically surveyed and analyzed various studies that have employed affective computing tools in the context of health care.

Technologies employed in emotion recognition

We performed our analysis according to the steps described in Figure 1. First, we searched the PubMed and Google Scholar engines for the terms "affective computing" and "affective technologies." Considering previous reviews on the topic2,6 and a previous PubMed search, presented in Supplementary Table S1, we focused on articles using the following methods of obtaining signals: 1) electrocardiography (ECG); 2) electroencephalography (EEG); 3) galvanic skin response (GSR)/skin conductance (SC); 4) thermal camera (TC); 5) bracelet; and 6) multimodal methods.

We acknowledge the significance of multimodal analyses and their potential to improve the precision of emotional recognition outcomes. However, our investigation focused specifically on the methodologies employed in monomodal research, allowing for a more targeted evaluation of methodological rigor without the confounding complexities introduced by integrating diverse device outputs in multimodal studies. This deliberate emphasis sought to circumvent methodological redundancies within individual studies, particularly given the heterogeneity of devices employed to capture analogous physiological signals. As a result, we selected only articles on monomodal techniques and evaluated their reproducibility and analytical quality using the framework proposed by Peng36 and our metrics, respectively.

Through the search results, we considered mainly, but not only, studies on Ekman's basic emotions (joy/happiness, fear, anger, sadness, disgust, and surprise).37 Of these, we selected 104 monomodal and multimodal articles. As already noted, the monomodal method analyses emotional event signals using a single input,38 while the multimodal method uses more than one source for emotion detection and recognition.39 Among these articles, we separated for analysis only those on monomodal techniques, comprising 61.54% of the studies: EEG (41.66%), ECG (26.66%), TC (16.66%), bracelet (8.33%), and GSR/SC (6.66%).

In this study, we considered three classifications for methods in terms of their invasiveness. Non-invasive methods are those in which the participant does not realize that they are being analyzed or monitored (e.g., video, audio). Low-invasive methods are those that still allow participants to remain comfortable with minimal stressors (ECG, EEG, and GSR). Invasive methods, which were not included in this study, are highly stressful and can interfere with emotional responses (computed tomography and electrocorticography).

Next, we analyzed the selected articles based on reproducibility and the number of participants. Reproducibility of results is the standard by which scientific claims are evaluated.36 The number of participants, or sample size, confers reliability to the study. The sample size corresponds statistically to the population and should be statistically significant.40

In many fields of study, there are examples of scientific research that cannot be fully replicated due to a lack of resources to enable its reproduction. In this sense, Peng36 has proposed major criteria to determine the reproducibility of a study ensuring scientific validation for its results. We assessed the reproducibility of the reviewed articles with a metric inspired by Peng.36,41

We considered the following items for evaluating the articles surveyed: number of participants; Peng's metric, which consists of a discrete metric that awards 1 point for the following requirements: provide (i) data, (ii) the code used, (iii) and the link between data and code36; and the result of the metric R, obtained as follows:

R   =   l o g n   +   p (1)

where n is the number of participants and p reproducibility (Peng's metric). The higher the value of R, the better rated the article. Finally, we present our results by ranking the articles in order of score.

An additional meta-analysis was carried out, using 64 articles on monomodal techniques, to determine which signal was best for each type of discrete emotion (considering only Ekman's big six). Due to the availability of statistical data, the accuracy metric was used to compare the studies. The highest average accuracy was used as a criterion for comparing the different monomodal signals.

Affective computing applied to health

We performed our analysis according to the steps illustrated in Figure 2. We searched for the terms "alexithymia affective computing," "affective computing pain," "neurodegenerative diseases affective computing," "affective computing disorder of consciousness," and "depression affective computing" on PubMed and Google Scholar (Supplementary Table S2). The search was limited to the period from 2010 to 2023. In view of the high number of results presented by the Google Scholar platform, we applied filtering using advanced search criteria: exact phrase and the use of at least one of the words defined in the search query. Use of these filters yielded a median reduction of 98% in the number of results, as shown in Supplementary Table S3.

We did not survey duplicate results present in both platforms. However, we did find a double result for neurodegenerative diseases and consciousness disorder, as described in the Impact of affective computing in health: gaps and insights from current research section.

When searching, we realized that the results for stress and depression would need to be refined further, due to the large number of articles not related to affective technologies being returned. Depression and stress per se are commonly related to affective states. Thus, we added the term "emotion" for a more targeted search that would yield studies focusing on emotions.

After the search, we selected articles for our analysis based on the titles and abstracts. Review articles, books, and research based on purely subjective data, such as questionnaires and self-assessments, were excluded. Next, we analyzed, based on our metrics, the studies that used affective computing techniques applied to health.

 

AFFECTIVE COMPUTING TOOLS: MONOMODAL AND MULTIMODAL METHODS FOR EMOTION DETECTION

Affective computing tools applied to health are based on particular responses or signals of the human body. One of the most common such signals is the heartbeat, captured by methods such as wristbands,42 gloves with sensors,43 or devices attached directly to the chest.44 Brain signals are usually captured using classical methods such as EEG45 or functional magnetic resonance imaging (fMRI).46 The skin conductivity reading method uses the mineral salts expelled by sweat, which conduct electricity, to identify variations in this parameter caused by emotions.47 In addition to these methods, those focused on capturing facial expressions are also widely studied. Pantic & Bartlett,48 for example, reviewed techniques for identifying facial expressions over a decade. In addition to broad representation in the literature, these methods can rely on immense datasets, such as EmotioNet, which contains one million facial expression images collected from the Internet.49 So much data requires specific studies.

Supplementary Figure S1 presents the devices and methods used to identify emotions explored in this review. The following section describes the main signals captured by each technology.

Monomodal studies in emotion detection: tools and techniques for physiological signal analysis

As noted above, our review was limited to studies using monomodal methods. Within these, we focused on five specific tools: ECG, EEG, SC, TC, and wearable devices (smart bracelets or wristbands).

Electroencephalography

The EEG records electrical activity in the brain, reflecting neurophysiological signals from various brain regions involved in perception and cognition. For instance, the frontal lobe is associated with thought and consciousness, the temporal lobe processes sensory stimuli such as smells and sounds, the parietal lobe integrates sensory information and manages object control, and the occipital lobe is related to vision.50 Fear, for example, activates the amygdala, a region essential for emotional processing51 which plays a role in various behaviors.52-54 In EEG studies, the signal-to-noise ratio (SNR) is a key metric, representing the relationship between signal power and noise power.55 Researchers typically examine the frequency, energy, and power spectra of different EEG bands.56 Some authors define five rhythmic frequency bands, while others consider six, with minor variations in their thresholds. The five main bands are: delta (0.5-3 Hz), related to deep sleep and exhaustion; theta (4-8 Hz), occurring during sleep and meditation; alpha (9-13 Hz), present in calm and unstressed states; beta (14-30 Hz), associated with active thought and work; and gamma (above 31 Hz), linked to focus, anxiety, and emotional stress.57,58 Studies on basic emotions have associated the gamma and alpha bands with happiness and sadness,59-61 while fear has been connected to delta, theta, alpha, and gamma waves in various brain regions.62,63 Continuous emotional models, such as valence-arousal, are more commonly related to EEG signals than discrete models.57,58 (For a comprehensive review of EEG and emotion detection, see Liu et al.57 or Rahman et al.58) One limitation of EEG is the presence of noise, common in electrical signal readings, which poses challenges for implementing machine learning algorithms in emotion recognition. Additionally, the signal acquisition process often requires tens to hundreds of electrodes and sampling periods lasting several seconds. This results in datasets with relatively few samples, which has led to a dearth of EEG data on cognitive neuroscience tasks.55

Electrocardiography

The emotional experience, triggered in the brain, leads to changes in heart rate through the autonomic nervous system (ANS). These changes can be detected via ECG, which captures electrical signals from the heart.24 ECG can also measure heart rate variability (HRV), a non-invasive marker of ANS activity.64 The ECG has been widely used in affective computing to assess heart rhythm variations, which indicate emotional responses. For instance, pleasant emotions increase heart rate,65 while HRV decreases in response to unpleasant emotions such as fear or sadness.64 The acceleration or deceleration of heart rate is linked to heart rate asymmetry (HRA), a promising marker for diseases affecting neuro-cardiac control.66 Positive heart rate responses lower blood pressure and muscle tension,67 whereas negative emotions raise the risk of coronary heart disease (CHD).68

Thermal imaging

Thermal imaging, particularly with infrared (IR) cameras, is a non-invasive method that measures skin temperature by detecting electromagnetic radiation in the 0.7-14.0 µm wavelength range.6 These cameras, often integrated into robots or smartphones, are widely used to study emotional responses by tracking thermal variations in facial regions, such as the eyes, cheeks, forehead, nasal tip, and mouth.69 Thermal imaging-based studies have grown significantly in the last decade, with over 2,500 articles indexed in PubMed, 970 of which were published in the last 3 years. However, publicly available thermal imaging databases remain scarce. Additionally, thermal profiles can vary significantly between individuals, even under controlled environmental conditions, complicating their interpretation.70,71 More research is needed to explore the potential of thermal imaging in diverse contexts and populations.

Skin conductance

SC, often measured via electrodermal activity (EDA), is a key indicator of emotional arousal.72 Electrodes measure the skin's electrical properties as sweat secretion changes, primarily from eccrine sweat glands, which are activated by emotional stimuli.72 Positive and negative emotions both influence SC responses.73,74 SC has been used in research on stress, anxiety, fear, emotion, and psychopathology, and is also employed in behavioral therapies such as biofeedback to help individuals regulate their emotional responses.2 SC measurements are commonly taken from the fingers, hands, and feet, which are the most responsive areas, while regions like the armpits and back show the poorest responses.72

Bracelet

Wearable devices, such as smart bracelets or wristbands, are increasingly used in affective computing to monitor and interpret emotional and physiological states. These devices often include sensors for heart rate, SC, and movement (accelerometers),75,76 capturing real-time emotional data.77

Emotion recognition using multimodal methods: a decade of research

Multimodal methods integrate various signals to capture a more comprehensive representation of emotions, often leading to higher accuracy.78,79 Efficient means can aid in emotion recognition, since emotion expression occurs throughout the body as a temporal arrangement of multimodal signals.39

In the last 10 years, there has been an increase in studies focusing on emotion recognition using multimodal methods, but these are still much less common. In a PubMed search limited to the period 2010-2023, we obtained 2,167 results for the expression "multimodal emotions." For comparison, our search for monomodal methods (limited to the same period) yielded 7,009 results for "emotions EEG," 5,228 results for "emotions speech," 3,749 results for "emotions skin conductance," 1,369 results for "emotions ECG," and 22 results for "emotions bracelet." Multimodal studies are not covered further in this review.

 

AFFECTIVE COMPUTING TOOLS FOR DIAGNOSING AND MANAGING EMOTIONAL COMPONENTS OF DISEASES

Emotions have long been left out of medical and scientific research. Now, however, there is mounting evidence that emotions - especially stress, anxiety, anger, and depression - are important factors with serious and significant implications on health.5 A cursory search of the PubMed platform highlights the increasing research interest in the role of emotions in health, with exponential growth in recent years (Figure 3).

Affective computing enables research and understanding of the relationship between human emotions and health. This, in turn, can offer useful insights for medicine. For Picard,5 the goal of new affective technologies for medicine should be to help medical professionals meet patients' health needs - both emotional and non-emotional - in a balanced, respectful, and intelligent way.

Luneski et al.2 published a review of the uses and advances of affective computing and its applications in medicine up to 2010. As an update of this review, we present below the main applications of affective computing in the field of health and the benefits of incorporating emotion recognition systems in each of them.

Methods for emotion detection in healthcare can be non-invasive (such as video detection of body movement and facial expressions, or speech) or low-invasive (such as EEG or ECG). Non- or low-invasive methods are always preferable, and sometimes necessary, such as to support a diagnosis of autism spectrum disorder.80

Among the methods, facial expressions are strongly correlated with emotions and represent a non-invasive approach.37,81-83

Diseases and disorders

Health is related to homeostasis, which is the body's state of balance. When this balance is destabilized, the normal functioning of the body can be affected, promoting the onset of diseases and disorders. Several factors can affect the balance of the body, among them emotions.1 There are many different types of diseases and disorders, each of which has an emotional basis or at the very least some emotional component.3

Affective computing tools can help identify the emotional components associated with diseases and disorders to aid in the prevention, diagnosis, and treatment of these conditions. We reviewed studies that used technologies to prevent, identify, treat, or predict diseases and disorders that have some relationship to affective states. Supplementary Tables S4-S9 list the selected articles.

Alexithymia

Autism spectrum disorder (ASD), or simply autism, is a complex disorder that includes several levels of impairment and may or may not incorporate the trait of alexithymia (difficulty in identifying, interpreting, and describing one's and others' emotions).84-86 Although this restriction is unspecific to ASD, it is associated with some of the disorders on the spectrum, serving as a diagnostic criterion.

ASD has significant heterogeneity and alexithymia, when present, has several levels of severity. One of the criteria for diagnosis of ASD is the reduced use of facial expressions. Besides difficulty recognizing emotions, individuals with alexithymia have decreased facial expression.80,86 Due to this trait, affective computing has the potential to help diagnose ASD.

Furthermore, current intervention techniques for children on the autism spectrum suggest that many of them can make progress in recognizing and expressing emotions when they learn emotions. Therefore, some tools from affective computing have also been used in the emotional education of persons with ASD.

Supplementary Table S4 lists the studies included in this review that used affective computing tools in the diagnosis and/or emotional education of people with alexithymia.

Pain care and the distinction of pain and emotions

Like emotions, pain can be detected by monitoring facial expression, body movement, electromyography,87 SC, pupil dilation, ECG, and other parameters. Detecting facial expressions that differ from emotion-related expression was among the methods best able to identify pain.88,89 However, pain is also associated with specific negative emotions, such as agitation and disgust.88,90 The Facial Action Coding System (FACS)37 also describes facial expressions of pain.

The detection of pain-related emotions may offer possible solutions for the treatment of chronic pain, in which emotional factors define the success of rehabilitation.91 Pain-related emotions are a barrier to recovery from chronic pain, and systems trained to detect emotions automatically may be a potential solution.91

Affective computing tools can be applied in monitoring,92 prevention,93 identification, and management of pain.

Supplementary Table S5 lists the studies included in this review which used affective computing tools in the diagnosis of pain.

Disorders of consciousness (DOC)

Patients with DOC, such as coma and vegetative state, suffer from reduced mobility and are generally unable to adequately express emotions.94 There is a paucity of studies that aim to differentiate emotions via EEG in individuals with these conditions.

Affective computing tools have been investigated as a potential tool to assess and manage emotional and psychological distress in patients with disturbed consciousness, such as those in a vegetative or minimally conscious state.95 These systems can help healthcare professionals track patients' feelings and behaviors over time, which would allow them to detect patterns that may indicate DOCs and could help provide more effective and earlier treatment. Affective computing technologies, such as facial expression recognition96 and physiological detection,97 can also be used to assess the emotional and physiological states of patients already in such states and provide appropriate interventions to improve their well-being. For example, affective computing can be used to design interactive environments or communication systems that can respond to patients' emotional cues and provide appropriate feedback or stimulation.98

Supplementary Table S6 lists the studies included in this review which used affective computing tools in the diagnosis of DOC.

Neurodegenerative diseases

People with neurodegenerative diseases have altered perception and socioemotional response.99 For example, patients with behavioral variant frontotemporal dementia (bvFTD) have a low resting state of emotional arousal and emotional blunting that may be verifiable via SC. The insula and cingulate cortex and the amygdala are involved in SC regulation and are areas affected by bvFTD.100

Studies highlight the relationship between SC and feelings of threat, suggesting it is a predictor of emotional responses to stressful life events.101 Compared to patients with Alzheimer's disease and healthy control subjects, patients with bvFTD - even though they exhibit a startle response - have low levels of SC.100

Emotion recognition could be a diagnostic aid for neurodegenerative diseases in their early stages. However, apathy, mobility difficulties, and changes in emotional regulation associated with normal aging make it difficult to identify emotional disorders when such diseases are already established. To address this challenge, the development of experimental protocols that assess emotional state through home-based affective computing technologies (e.g., smart homes) has been proposed.102

Smith et al.103 state that affective computing can become an important care delivery tool for late-stage affective and cognitive disorders. They can serve to identify late-life cognitive and mood disorders, including depression and Alzheimer's disease. Affective computing technologies can offer objective biomarkers and tools for early detection, monitoring of treatment response, and tracking disease progression, as well as provide a more comprehensive understanding of patients' daily lives. Pontieri et al.104 compared facial emotion recognition performances between patients with progressive supranuclear palsy (PSP) and a large group of patients with Parkinson's disease (PD), examining the different encoding thresholds for emotional faces in PSP and PD.

Supplementary Table S7 lists the studies included in this review which used affective computing tools in the diagnosis of neurodegenerative diseases.

Depression

Prolonged sadness can lead to depression, which in turn is one of the most common mental disorders and is a leading cause of disability worldwide. This disorder affects people of all ages, including older adults. Geriatric depression has become a public health problem. Its effect on health increases when it is comorbid with a chronic medical condition.1

Affective computing can provide tools for the identification of depression, allowing a more objective analysis of its different levels of severity. Such technologies can also serve as a support tool for the treatment of patients.

Supplementary Table S8 lists the studies included in this review which used affective computing tools in the diagnosis and/or treatment of depressive disorder.

Stress

Stress is a state of emotional arousal that promotes the onset of disease.1 Therefore, it is a risk factor for poor health, necessitating better measurements.

Affective computing tools can assist in identifying a patient's stress levels, providing objective data for the patient themselves and health professionals, and can play a crucial role in the diagnosis of stress by integrating emotional recognition technologies and biometric data analysis to monitor physiological and behavioral reactions in real time. Recent studies, such as that by Wang et al.,105 have shown that analysis of facial expressions, vocal tone, and physiological patterns, such as heart rate and HRV, can help identify stress states more accurately and efficiently. These technologies allow for early and personalized interventions, enhancing the effectiveness of stress diagnosis and treatment while providing valuable insights into individuals' mental health.105

Supplementary Table S9 lists the studies included in this review which used affective computing tools in the diagnosis of stress.

Irritable bowel syndrome

IBS is a multifactorial condition which causes visceral hypersensitivity and pain. IBS is related to emotional states of anxiety, depression, and stress reaction and exposure. Such intense emotional conditions lead to an imbalance in the relationships between the ANS and the hypothalamic-pituitary-adrenal axis, which alters the homeostasis of the gut biota, leading to development of IBS.106

There are no studies that demonstrate or propose the use of emotion recognition as a tool to aid in the recognition and/or treatment of IBS; however, emotion monitoring may be useful, as is already being done in bipolar disorder.107,108 As noted by Pellissier & Bonaz,106 IBS is a multifactorial disorder involving biological, psychological, and social stimuli. Monitoring emotional state and recognizing stress, one of the main triggers of gastrointestinal symptoms, may assist in the control of the syndrome.

Affective computing can help detect patterns in patients' symptoms and behaviors, allowing for a more accurate diagnosis of IBS. It can also be used to provide personalized care and treatment plans tailored to the needs of the individual. In addition, it can help to better understand the psychological aspects of IBS, as well as its social and cultural impacts. For example, affective computing could be used to track a patient's emotional responses to stress,109 as well as collect, analyze, and interpret data related to the expression of IBS symptoms, including physiological and emotional information.110 These data could be used to identify patterns and predictors of IBS flares and to inform diagnosis and treatment. Finally, affective technologies could be used to monitor patients' progress and response to treatment.111

Although there are numerous possibilities for the practical application of affective computing in the diagnosis and emotional treatment of people with IBS, we did not find any studies on this topic.

Smart homes and environments in emotional well-being and health care

The number of studies focusing on developing smart homes - monitored environments enriched with non-invasive sensors located in various rooms - and other smart environments has grown. Among the many uses of these spaces, assisted living for older adults is one of the most studied and growing applications. Smart homes had the initial purpose of providing security and energy savings; however, their focus has now turned to health care, wellness, and comfort to assist people with disabilities, older adults, and people with reduced mobility (e.g., wheelchair users). The authors of a critical review on smart environments emphasize that these environments need to provide emotional well-being and, therefore, should provide entertainment and social experiences.112

Another emerging use of smart technology in health care systems is the monitoring and care of patients with psychiatric disorders, such as bipolar disorder. For example, Grünerbl et al.107 proposed using a mobile phone-based system that monitors the behavioral aspects of patients with the condition, while Prociow et al.108 aim to provide an early warning system mediated by wearable and environmental sensors. The signals captured by these sensors offer clues about the patient's emotional state, which may lie between depression, normality, and mania, allowing the psychiatrist to follow up. Monitoring mediated by such systems would make life easier for patients with bipolar disorder and support their treatment by providing vital information to their psychiatrist. Furthermore, this information ensures better decision-making, supporting implementation of any necessary therapeutic measures, and may prevent incidents, including suicide.107,108

Additionally, there is a wide range of work focused on the use of emotional state recognition47,113 for behaviors linked to fatigue,114 stress, drunkenness, and other dangerous behaviors of motor vehicle drivers.115,116 Several studies have also focused on drivers' emotions to prevent automobile accidents and thus promote the health and safety of road users.117

Recognizing emotions in the real world

There are considerable differences between real-world (or "in-the-wild") environments and controlled (laboratory/posed) environments. According to Tzirakis et al, these may include lack of noise or reverberation, unobstructed visibility, poor lighting, and varied/non-uniform backgrounds. Participants in controlled environments are also restricted from moving as freely as they would in the real world. Similarly, posed faces display a limited range of expressions and lack the subtle nuances that naturally occur in realistic, spontaneous environments.118,119 All these issues interfere with the efficiency of emotion recognition methods.118

Applying emotion recognition methods in uncontrolled environments tends to achieve much lower accuracy and effectiveness than in controlled environments.120 However, several studies have achieved significant results.121,122 For example, Liu et al.122 achieved 66.9% accuracy and an F1-score of 40.8% in classifying images extracted from the Aff-Wild2 dataset of Kollias & Zafeiriou,123 which combines the ResNet and Bidirectional Long Short-Term Memory (BLSTM).

There have also been specific studies aimed at circumventing problems brought about by real-world emotion recognition,124 some of which aimed to address particular issues such as recognizing facial expressions with occlusion125-128 and noisy environments.129 Studies based on multimodal systems, which combine face (video), text (written), and speech (acoustic, words) data, are able to deal with the noise and occlusions characteristic of in-the-wild data.130-133

Mollahosseini et al.34 point out the scarcity of in-the-wild datasets with labeled images and survey the principal available datasets. They go on to propose AffectNet, a set of 1 million in-the-wild facial images of which 450,000 have been manually annotated into eight discrete and valence-arousal emotional categories.

 

RESULTS

Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes

We found a substantial amount of studies which scored low in our evaluation, as shown in Table 1.

Table 1. Study quality according to Peng's metric36 and our metric (equation 1)
Subjects Data Code Link Peng's36 metric Our metric Reference
5 0 0 0 0 0.698 Wang134
5 0 0 0 0 0.698 Lin135
6 0 0 0 0 0.778 Pan136
10 0 0 0 0 1 Yu137
10 0 0 0 0 1 Li & Lu61
10 0 0 0 0 1 Boccanfuso138
12 0 0 0 0 1.079 Wei139
14 0 0 0 0 1.146 Jerritta140
14 0 0 0 0 1.146 Alsolamy & Fattouh141
16 0 0 0 0 1.204 Petrantonakis & Hadjileontiadis142
17 0 0 0 0 1.23 Goulart143
19 0 0 0 0 1.278 Jalilifard144
19 0 0 0 0 1.278 Shahabi & Moghimi145
20 0 0 0 0 1.301 Zhang146
20 0 0 0 0 1.301 Murugappan147
20 0 0 0 0 1.301 Taran & Bajaj148
20 0 0 0 0 1.301 Khare & Bajaj149
21 0 0 0 0 1.322 Mehmoo150
21 0 0 0 0 1.322 Mehmood & Lee151
21 0 0 0 0 1.322 Raheel152
25 0 0 0 0 1.397 Dissanayake153
25 0 0 0 0 1.397 Guo154
25 0 0 0 0 1.397 Shu75
25 0 0 0 0 1.397 Yu & Tapus155
26 0 0 0 0 1.414 Lin156
26 0 0 0 0 1.414 Basu157
30 0 0 0 0 1.477 Bhatti158
30 0 0 0 0 1.477 Zhuang159
33 0 0 0 0 1.518 Tarvainen160
35 0 0 0 0 1.544 Kolli161
36 0 0 0 0 1.556 Mikhail162
44 0 0 0 0 1.643 Cruz-Albarran163
50 0 0 0 0 1.698 Quiroz164
59 0 0 0 0 1.77 Zhang76
59 0 0 0 0 1.77 Cui165
60 0 0 0 0 1.778 Selvaraj166
67 0 0 0 0 1.826 Nayak167
75 0 0 0 0 1.875 Agrafioti168
84 0 0 0 0 1.924 Xun & Zheng169
154 0 0 0 0 2.187 Wan-Hui170
25┼ 1╬ 0 0 1 2.397 Xianhai171
254 0 0 0 0 2.404 Wu172
26 1╬ 0 0 1 2.414 Ferdinando173
26 1 0 0 1 2.414 Chen174
26 1╬ 0 0 1 2.414 Nguyen175
27 1╬ 0 0 1 2.431 Ali176
27 1╬ 0 0 1 2.431 Wei177
32 1╬ 0 0 1 2.505 Vijayan178
32 1╬ 0 0 1 2.505 Zamanian & Farsi179
32 1 0 0 1 2.505 Chen180
391 0 0 0 0 2.592 Jing181
391 0 0 0 0 2.592 Xu & Liu182
77 1╬ 0 0 1 2.886 Brás183
975 0 0 0 0 2.989 Alghowinem184
28 1 1 0 2 3.447 Goulart185
32 1╬ 1 0 2 3.505 Balan186
2,340 1§ 0 0 1 4.369 Sarath187

The higher the value, the higher the reproducibility of the study and the larger the sample size.

┼ The samples correspond to 25 days instead of participants.

╬ Public data.

§ The data refers to an image database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible camera and an infrared (IR) thermal camera (TC), with lighting provided from three different directions.

Considering only reproducibility scores, which range from 0 to 3, most articles scored between 0 and 1. Only two studies achieved a score of 2, as they provided both the data and codes used. However, most studies with available data were due to the use of already public data.

Taking into account only the number of participants, the population sample of the articles was not significant. Most of the monomodal studies analyzed (34.48%) had samples of fewer than 30 individuals and only 12.06% had more than 100 participants; and none had a population greater than 1,000 participants. Of the reviewed studies, 72.41% performed their analyses on a population of fewer than 5 participants.

Considering both reproducibility and sample size, we obtained our metric (equation 1), whose distribution is shown in Figure 4. The figure highlights the low quality of the articles overall, implying low reliability of the results. Only one included study obtained a score above 4, and two obtained a score above 3.

Sarath et al.187 had the best result, with a score of 4.369. This study sought to examine the emotions hidden in the face, including stress and anxiety, through facial temperature measurement.187 Balan et al.186 obtained a score of 3.505. The goal of their study was to define accurate methods for classifying the level of fear. In future research, the authors intend to develop a phobia treatment system based on gradual exposure to virtual reality, which automatically determines fear EEG levels and biophysical data.186 Goulart et al.185 achieved a score of 3.447. The purpose of their research was to propose an experimental design for analysis five emotions (disgust, fear, joy, sadness, and surprise) in children aged 7 to 11 years, by means of changes in facial emissivity detected by infrared thermal imaging (IRTI).

Two among the three highest-scoring articles in our review established relationships between emotions and health, strengthening the argument for the applicability of affective computational tools in the health sciences. Sarath et al.187 discuss the use of a computational tool that identifies emotions, among them stress and anxiety, two factors that can increase the risks of sudden cardiac arrest, a leading cause of mortality and morbidity in affluent societies.188 Balan et al.186 analyzed fear levels with the future goal of creating a tool for the treatment of phobia.

Finally, based on the accuracy of the classification of emotions in the studies surveyed, we checked which signal was most effective in classifying each of Ekman's six basic emotions. Supplementary Figure S2 shows the average percentage across studies for each signal used.

We observed that some signals served the purpose of identifying emotions better than other methods. In addition, we identified that specific signals recognize certain emotions more efficiently than others. We were thus able to associate each emotion with a capture method or physiological signal that is most efficient at identifying it.

Out of the 109 articles initially analyzed, 64 were monomodal, employing EEG (n=26), ECG (n=17), TC (n=11), accelerometry (n=6), and GSR/SC (n=4). The results showed that TC had the best average accuracy for detection of happiness (83.48%), while ECG was the best for detection of sad (79.83% average accuracy). Finally, smart bracelets had the best average accuracy for identifying fear, anger, disgust, and surprise, at 82.57, 89.30, 84.91, and 88.01%, respectively. These results show that the three best monomodal tools for studying basic emotions are bracelets or wristbands, thermal imaging, and ECG.

One caveat is warranted: accuracy is not the best statistical metric for model validation, as will be further addressed in the discussion. Therefore, these results are still only indicative and require further studies for actual validation.

The impact of affective computing in health: gaps and insights from current research

There is a limited amount of research on the application of affective computing tools to health. While some studies have linked these areas, little research has focused on capturing and identifying emotions and applying this to the prevention and treatment of diseases. Much research had used affective technologies to predict disease, propose treatments, or teach patients to recognize and manage emotions without, however, detecting these emotions. In the case of alexithymia, for example, some research focuses on emotional education. As for pain, depression, and stress, the disorder or condition itself may, in some studies, become the object of emotional capture and identification.

Supplementary Table S10 presents the results of our search. The largest number of studies retrieved concerned neurodegenerative diseases. We believe this was due to the lack of a more targeted filter. The filters used allowed for a large volume of articles and reviews on the topic to be retrieved by the Google Scholar search tool. Few of these searches applied affective computing tools to neurodegenerative diseases. One of the articles31 appeared in the results for disorder of consciousness. Since the focus of the study was PD, we decided to keep it among the results for neurodegenerative diseases.

IBS showed only one result in Google Scholar and none in PubMed. We chose to keep this disease in our discussion because we realized, in our rationale, that there is a potential research field for affective computing tools to aid in treatment of this syndrome.

There were fewer results for depression and stress. We believe this was due to the use of filter 3, in which we added the word emotion to the search query. This was done because the words stress and depression, by themselves, result in retrieval of publications broadly related to affectivity, not specifically emotions. Targeting allowed for targeted results focusing on emotions. This term was not added to the other search strings because, conversely, not targeting emotions provided results related to the field of affectivity in general.

In our final sample, most studies focused on the recognition of depressive disorders. Wang et al.189 used sensors to assess gait. Dham et al.29 used multimodal techniques to recognize depression from a combination of audio, visual, and text analysis. Liu et al.,28 He & Cao,30 and Hansen et al.33 used classification and measurement tools and facial recognition to identify depression through speech. Guo H. et al.32 reported a method for recognizing depression through facial expressions. Huang et al.190 compared several machine learning methods to identify a model that most accurately predicted suicidal ideation and depression level in school-aged adolescents.

After reviewing the articles, we applied our quality metric (Table 2) to identify replicability and methodological reliability.

Table 2. Study quality according to Peng's metric36 and our metric presented in equation 1
Subjects Data Code Link Peng's36 Our metric Reference
8 0 0 0 0 0.903 He191
10 0 0 0 0 1.000 Liu & Jiang192
11 0 0 0 0 1.041 Varotto193
12┼ 1╬ 1 0 2 3.079 Perez-Toro194
14 0 0 0 0 1.146 Pan195
15 0 0 0 0 1.176 Hosseini196
15 0 0 0 0 1.176 Moon & Qu197
18 0 0 0 0 1.255 Huang94
18 0 0 0 0 1.255 Zolyomi & Snyder198
20 1 1 1 3 4.301 Prossinger199
20 0 0 0 0 1.301 Bourvis200
22 0 0 0 0 1.342 ArulDass & Jayagopal201
24 1╬ 0 0 1 2.380 Dham29
30 0 0 0 0 1.477 Akbulut202
42 0 1 0 1 2.623 Hansen33
50 1 0 0 1 2.699 Sparrow203
52 0 0 0 0 1.716 Filippou204
55 0 0 0 0 1.740 Moretta205
65 0 0 0 0 1.813 Goldstein27
74 1 1 1 3 4.869 Atee206
77 1 1 1 3 4.886 Kang207
83 1 0 0 1 2.919 Garcia-Cordero208
85 0 0 0 0 1.929 Wu209
85 1 0 0 1 2.929 Campbell210
95 0 0 0 0 1.978 Wang211
104 1 0 0 1 3.017 Guo32
109 1 1 1 3 5.037 Birba212
184 0 0 0 0 2.265 Liu28
195 1 0 0 1 3.290 Pontieri104
300 1╬ 0 0 1 3.477 He & Cao30
873§ 1 1 0 2 4.941 Caldo213
10,234 0 0 0 0 4.010 Huang190
131,025|| 1╬ 0 0 1 6.117 Dar31

The higher the value, the higher the reproducibility of the study and the larger the sample size.

┼ The samples correspond to 12 hours of audio instead of participants.

╬ Public data.

§ The samples correspond to 873 URLs instead of participants.

|| The samples correspond to 131,025 seconds instead of participants.

We highlight the seven articles with scores above 4, out of a total of 34. First among them was Dar et al.,31 with a score of 6.11. The researchers applied the BCI emotion recognition system to patients with OCD. This study was cited in nine other articles,214-222 as a reference on the use of affective computing tools in health care. Its results show that the proposed BCI system can be a promising tool for emotion recognition in patients with DOC.

In second place, Birba et al.,212 with a score of 5.03, used a multimodal combination of behavioral, neuroanatomical (voxel-based morphometry [VBM]), and resting-state functional connectivity (rsFC) (fMRI and high-density EEG derivatives [hd-EEG]) measures to examine key signatures of naturalistic understanding of social actions and speech in patients with PD and bvFTD, compared to healthy controls and patients with Alzheimer's disease.

In the third position was Caldo et al.213 with a score of 4.94. In this study, the researchers developed a machine-learning algorithm with the goal of distinguishing discrete emotional fingerprints from web pages related to back pain.

Kang et al.207 came fourth, with a score of 4.88. Their paper presented the K-EmoPhone, a set of multimodal and physiological data collected from participants' smartphones and self-reported affect states, including emotions, stress, attention, and task interruption, acquired by the experimental sampling method.

In fifth place, with a score of 4.86, Atee et al.206 described a new method, called PainChek, aimed at assessing and monitoring pain, as well as profiling patients and synchronizing data.

Sixth was Prossinger et al.,199 with a score of 4.30. In this study, the researchers use AI to analyze facial expressions and determine the distances between the affective states Neutral-Pain, Neutral-Pleasure, and Pleasure-Pain.

Finally, in seventh place with a score of 4.01, Huang et al.190 proposed a method of emotional diagnosis through the recognition of facial expressions.

The two highest scores were achieved by studies on neurodegenerative diseases. The highest number of studies with scores above 4 was research that related affective computing to pain. Of the seven articles ranked highest, three were about pain. Depression and stress had one article each with a score above 4.

Most of the included studies (44.11%) scored less than 2.0 points, including all studies on DOC and alexithymia. Of the seven research papers on depression, five had scores between 1.0 and 2.0. Only one article on depression and two on neurodegenerative diseases were between these minimum limits.

 

DISCUSSION AND CHALLENGES

We begin by presenting a discussion on affective computing research, based on our analysis. Next, we evaluate studies on the application of affective technologies to the prediction, diagnosis, and treatment of various diseases. Finally, we highlight the challenges faced by the field, as identified in our investigations.

Technologies employed for emotion recognition: shortcomings

The recognition of emotions is routinely carried out in our lives and facilitates interpersonal interaction, physiological reactions of emotional origin, and understanding of mental states. The homeostasis of the human body is influenced by physiological responses to the emotional context, and its imbalance can trigger diseases. Many studies reveal that emotions such as anger, anxiety, depression, and stress are significant medical factors.5

In recent years, an exponential growth of 233.54% in the study of emotions and their relationship to health and medicine has occurred (Figure 3). This growth reveals the scientific interest in the topic as stated by Consedine & Moskowitz.223

However, during our literature survey, we were faced with a systematic lack of concrete health applications of emotion detection techniques. For example, few articles used the term "affective medicine" explicitly. Despite the scarcity of studies as focus on the application of computational tools applied to affective medicine (described in the Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes section), there is a great potential and even a need to develop methods to capture human emotions and apply this to health. Affective computing emerges to provide technological means that capture and identify emotions and thus help professionals provide emotional support in the treatment of an illness.5

Despite the significant growth of studies exploring the relationship between emotions and health, as well as the promising potential of this field, several critical deficiencies persist in the study of emotions, their detection, and their application in health contexts. This review has identified that current technologies developed for emotion detection remain inadequate for practical implementation in human health.

In this work, we systematically identified gaps and evaluated existing studies by employing an evaluative scale inspired by the framework proposed by Peng.36,41 To address these challenges, we developed a metric grounded in Peng's methodology, which is widely recognized in the literature. While not without limitations, this metric provides an objective tool for assessing the methodological quality of the reviewed studies, surpassing the constraints of exclusively qualitative analyses.

Our findings reveal that many studies fail to meet essential criteria for reproducibility and scientific rigor. Common shortcomings include insufficient sample sizes and the lack of consistently validated methods. These limitations underscore the need for more robust methodological frameworks to ensure reliable and impactful advancements in the intersection of emotional recognition and health research.

Overfitting and underfitting are phenomena that occur during the training of neural networks. Overfitting occurs when a statistical model effectively fits the previously observed data set but fails to accurately predict new results. One of the causes of this phenomenon is sample size. When the training set is too small or has less representative or very noisy data, the results are compromised. Larger samples more accurately represent the value of the population, while smaller samples can present a statistically non-significant result. More heterogeneous studies require a large sample size to obtain accurate results.224 On the other hand, underfitting occurs when the model is not complicated enough and focuses too little on the training data.225 The problem with underfitting is that training is insufficient and learning accuracy is low.226 As a result, the model can neither fit the training set nor generalize well to new data.

In our analysis, we included the sample size criterion to identify one of the problems with validating research into affective computing. The majority of studies in the area present scarce data, compromising their results with overfitting. In turn, we excluded gender and age to avoid underfitting our analysis. Both influence the generalization of the data and could compromise our results, since some studies did not provide the age of the participants, others focused on very different age groups and still others on a single gender.

Among the monomodal articles analyzed, most have methodological inconsistencies or gaps, which prevent analysis of the reported tools. The lack of access to data and/or the size of the population jeopardize the quality of the results.

Additionally, studies - as a consensus - use accuracy as the metric to validate the trained classificatory methods for the various machine learning models used. However, the accuracy metric is insufficient to validate methodological reliability because it is subject to biases, such as when there is an imbalance between classes; in fact, it is one of the worst possible metrics for validation purposes. Among the metrics adequate for such a task would be the F1-score and the combined analysis of metrics such as specificity, accuracy, precision, and sensitivity, which were mostly uncalculated.

In relation to gaps highlighted by the proposed metrics, evidence of the low reliability of the results highlights the need for more general studies before their implementation in the health field, and justify why emotion recognition methods are not currently applied to health. What would explain this low study quality remains to be determined.

Affective computing technologies in healthcare: focus on neurodegenerative diseases and emotional diagnostics

As raised in the Limitations in emotion recognition studies: low reproducibility and insufficient sample sizes section, several potential uses of affective computing in healthcare can be explored. We highlight the development of methods for monitoring and tracking patients, as has already been applied to individuals with psychiatric conditions such as bipolar disorder,107,108 allowing medical specialists or psychiatrists to follow patients' progress and avoid incidents, technologies that can be improved and applied to other disorders such as schizophrenia. It also has great potential in helping patients with DOC94 who are otherwise partially or completely unable to communicate.

Despite the potential benefits, we find that, to date, little research had applied technologies to identify emotions in the context of health. Many studies used devices and tools such as EEG, ECG, etc., without focusing on emotion detection. Comparatively, the amount of research relating neurodegenerative diseases to affective computing, with the use of computational tools to analyze the emotional state of people with Alzheimer's disease, Parkinson's disease, and ALS, reveal the importance of this area of study for health. On the other hand, the lack of practical studies on the benefits of affective technologies in the diagnosis and/or treatment of emotional states of people with IBS demonstrates the need to invest in more research focused on affective computing applied to health. We also highlight studies on the detection of depression through voice. According to our surveys, most affective computing research focused on depression uses voice capture tools to diagnose the disorder.

Our analysis of methodological quality of research on affective computing applied to health showed that studies on neurodegenerative diseases performed best. We highlight the work of Dar et al.,31 which has become a reference in the application of affective computing tools in the care of patients with neurodegenerative diseases. This is a particularly promising area, as affective technologies can aid in the recognition and emotional assessment of people with PD, ALS, and dementia.

Research on the application of computational tools in the diagnosis of pain and related emotions also shows promise. Studies have shown good results. This area of research can lead to many advances in the diagnosis and treatment of pain.

There is a lack of qualitative research on the applicability of affective computing in the emotional diagnosis of people with disorders of consciousness. We recognize that this is a difficult area, but it is also an important field in which more studies are sorely needed to ascertain whether affective technologies can be used in the emotional assessment of people with DOC.

Having verified the importance of the applicability of affective computing to health, we highlight the need for more accurate studies. We suggest the use of a text mining tool to obtain more accurate results, with the classification of the most relevant and most necessary themes.

Overcoming challenges in emotion recognition for medical and real-world implementation

Based on the findings of our literature review, we established a methodological roadmap in the format of challenges that need to be overcome (Figure 5). These challenges represent the deficiencies observed in the process of recognizing emotional states which must be solved before adequate, reliable tools can be obtained for medical and general use.

The first challenge is to obtain a gold standard. The complexity related to the concept of emotion itself is an obstacle to a precise definition. There is also great difficulty in establishing a gold standard in interpreting emotions, given the impossibility of directly measuring an individual's emotional state.16,80 Consequently, there is a lack of standardization across studies, as reported by Mauss & Robinson.16

The affective computing literature is beset by a broad lack of a solid foundation. Striking results are validated by subjective elicitation methods. Several studies have questioned the reliability of using self-report questionnaires to assess felt emotions, depositing undue trust in emotion elicitation methods which may partially assure the triggering of the expected emotion.

This aspect warrants particularly careful analysis, because confidence in study results is essential to establishing standards and verifying research integrity. Regardless of its cost, invasiveness, or difficulty, this standard should allow identification of the emotion being felt within a current classification, usually a discrete model.

It can therefore be said that the biggest challenge in the field of emotions is to establish gold standards, whether for emotional models, elicitation methods, detection standards, or interpretation of results. A gold standard is essential for the standardization and reliability of analyses, both to make progress and to ensure safety in the use of new technologies, as well as to ensure future uptake and use by health care providers. Such standards will facilitate the validation and comparison of proposed methods and tools and will allow the construction of curated databases for different signals.

The establishment of a gold standard in affective computing necessitates the implementation of cross-validation with biometric methods. Integrating technologies such as EEG and fMRI is a promising approach for validating emotional recognition algorithms, offering a more precise and robust validation process. These methods provide a deeper understanding of the neural mechanisms underlying emotional responses, facilitating the development of more accurate models for emotion recognition.227 Moreover, incorporating physiological measures, including heart rate and GSR, provides objective references that enhance the reliability of emotional analyses by enabling data triangulation between physiological signals and computational models. This multimodal approach serves to strengthen the validity of the emotional data captured, ensuring a more comprehensive understanding of affective states.

Equally crucial for the establishment of a gold standard is the development of collaborative benchmarks and clear evaluation metrics. The creation of widely accepted benchmarks requires the concerted efforts of interdisciplinary teams, including experts from computer science, psychology, and other related fields. This collaborative approach is vital for ensuring that benchmarks reflect a broad range of real-world applications. Noteworthy initiatives, such as the Emotion Recognition in the Wild project, aim to bridge the gap between technology and real-world scenarios by providing explicit evaluation metrics, including precision, recall, and rates of false positives and false negatives. These metrics are indispensable for validating and refining emotional recognition systems, thereby enhancing their practical applicability in diverse contexts.227

The second challenge is related to elicitation methods. The reliability of emotional stimuli as a method of validation encounters a barrier in cultural diversity. Psychological studies have shown that the primary expressions of emotions are not culturally universal,228 confirming a cultural influence on emotional responses. Emotional expressions tend to be very different between Western and Eastern populations.229 However, automatic facial expression recognition systems disregard these differences, analyzing and recognizing them universally in different cultures.228 Most elicitation databases focus on Western people with different characteristics than Eastern people. Larger and more diverse emotion databases are needed to overcome this.229 Additionally, as noted in Supplementary Table S11, constructed datasets generally use self-validation methods (as listed in Supplementary Table S12) to validate the classified emotions, i.e., the labels of each emotion class. The fact that they are based on self-reported methods places the reliability of the data into question.

The third challenge in validating emotional recognition systems lies in obtaining a comprehensive diversity of signals, including physiological, behavioral, and facial expressions. These signals must be less invasive, cost-effective, and practical to collect, ensuring greater applicability in real-world and clinical contexts. While maintaining control over the research environment is crucial for ensuring data quality and consistency, there is an increasing need to broaden the representativeness of the collected data.

Striking a balance between rigorous control and representativeness is fundamental. Excessively granular data may introduce noise and reduce precision, whereas overly restrictive datasets risk bias and limit the generalizability of developed models. To address this challenge, we propose aligning data collection protocols with established validation standards in AI. These standards advocate for controlled, robust methodologies while promoting the diversity necessary to capture the complexity of human interactions.

Once multiple signals have been collected and classified with reliability equivalent to the gold standard, it becomes possible to explore which signal (or combination of signals) is most effective for emotion detection. This decision should carefully balance reliability, cost, and invasiveness. The answer will inevitably depend on the application context and specific circumstances in which the method is employed, highlighting the importance of adaptive and dynamic solutions.

The fourth challenge is in capturing and combining these signals. A particular combination of signals (multimodal methods) allows more efficiency in recognizing emotions. Parallel to evaluating the signals obtained, one must consider which computational methods are more efficient in performing this automated classification. Several machine learning methods can be employed to obtain reliable tools, with the use of reliable data.

The fifth challenge is related to datasets. We surveyed the main datasets (Supplementary Table S11) and found that emotions are induced effectively in the laboratory,230 which makes it feasible to use emotional datasets generated in laboratory environments. However, lack of generalizability is a problem in the datasets found in the literature. The fact that these data originate from laboratory settings, where images of individuals are captured in the same position, with standardized backgrounds and no visual obstructions, is not the main issue, but rather the homogeneity of the participants, who often belong to similar age groups, ethnicities, and nationalities. This limitation in participant diversity restricts the applicability of the results to a broader and more heterogeneous population. Furthermore, we emphasize the need for a more robust dataset for quantitative validation of study outcomes. Proper sample size calculation is crucial in quantitative research, as it directly influences the statistical validity and reliability of the inferences drawn from the data. This calculation takes into account factors such as margin of error, confidence level, and expected population variability. For studies aiming to represent broad and heterogeneous contexts, sample sizes between 400 and 1,000 individuals are often recommended, assuming a 5% margin of error and a 95% confidence level, as suggested by Krejcie.231 However, the studies reviewed in this analysis had significantly smaller sample sizes, ranging from eight to 300 participants. This reduced sample size may limit the generalizability of results and undermine the identification of more robust patterns, particularly in areas as complex and multifaceted as emotional and behavioral health.24

The sixth challenge is the application of machine learning in emotional recognition for health. Despite significant progress, this technology still faces barriers in emotional classification that need to be overcome.232 As well raised by Luneski et al.,2 affective medicine, linked to affective computing, has gaps in its applications waiting to be filled. These include studies focused on the prevention of emotion-related diseases. We found that emotion recognition technologies have made inroads in healthcare, especially in helping patients with neurological and neurodegenerative diseases such as dementia, bipolar disorder, autism, and disorders of consciousness. However, it is still far from being widely and efficiently used to aid diagnosis, prevention, or maintenance of well-being.

There has been little research into developing personal health systems incorporated into home-based health care systems, as corroborated in the study by Lee & Kim.112 The authors state that research and implementation of monitoring systems that consider the emotional well-being of older adults in smart homes are still scarce. We did find research focusing on late-life depression, but there remains a need to ascertain whether affective technologies can be used to improve the quality of life of older adults.

Finally, the seventh challenge is the difficulty of capturing and identifying signals in the real-world setting. In the real world, data acquisition is subject to background variation, occlusion, and movement (for images); noise, a greater variety of positions, angles, movements, and facial expressions and gestures; and real-time implementation issues, among other adversities.118 We can only achieve an effective, applicable method in the real world when these challenges are overcome.

The main barrier to emotion recognition in real-world environments stems from the lack of control over the factors that make up such an environment. As a result, most studies focus on creating and implementing emotion classification methods in highly controlled (laboratory) environments, which achieve considerable effectiveness, but cannot be directly extrapolated to real-world settings.

Few such studies have real-world health applications; most have focused on developing algorithms. There has been no validation or application to diagnosis, patient monitoring, etc. A scarcity of in-the-wild datasets with labeled images,34 necessary for the development and validation of recognition systems, is also noted.

Calibration for the real world is essential for implementation of affective technologies across diverse health care settings, such as home care, hospital practice, and the like. The prospects for applications of emotional state identification are broad for the near future, but the importance of further studies in real-world applications cannot be overstated.

To overcome these and other challenges, we highlight the importance of additional studies focusing on human emotions seen through the prism of affective computing and affective medicine. There is a pressing need for development of emotion databases that are more inclusive: comprehensive in terms of ethnic and age groups, including a large volume of participants, and, in particular, collecting real-world, heterogeneous data, labeled to allow future analysis using parameters with better-balancing refinements, such as the F1-score method. Note, also, that it is necessary to start by establishing a gold standard so that the next steps can then be carried out.

The challenges detected highlight and justify the dearth of literature. There is no solid evidence base on which to build further studies safely and reliably, including for health applications.

 

CONCLUSION AND PERSPECTIVES

The two-way relationship between emotion and health is evident and warrants greater attention in scientific research. The use of technology for emotion recognition can serve health in different ways, including aiding in the diagnosis and treatment of diseases and monitoring of patients.

Affective computing tools - i.e., those using the machine's ability to capture, analyze, and identify emotions - have the potential to aid in prevention and health care. However, there are great challenges to the real implementation of affective technology to support health and affective medicine.

In this review, we highlighted the most neglected methods for capturing heartbeats, brain signals, and skin conductivity. It is impossible to qualify the results and identify the best tools for emotional identification due to the methodological inconsistencies identified in the studies involving affective computing technologies. In addition to these inconsistencies -non-reproducibility and low number of participants - we were able to list seven technical challenges that are the main causes for the unfeasibility of these methods for emotion recognition in general, as well as specifically in health.

Among the challenges are the lack of a gold standard and poorly curated classes that make it difficult to reliably implement AI methods, which are based on pattern recognition. Consequently, development of systems applicable to real life, such as smart homes, robots, clinics, etc., is hampered.

We present three broad suggestions that can help overcome these challenges. First, conduct studies focused on obtaining a gold standard. From a well-established gold standard, it is possible to move forward and overcome the challenges presented in Figure 5. Second, conduct studies with larger sample sizes, greater diversity in age and ethnicity of participants, and in less controlled environments, using replicable methodologies and making data and methods available, as established by Peng,41 thus avoiding bias, overfitting, and promoting the generalization of models with greater reliability. Finally, we suggest further exploration of the potential use of emotion detection in healthcare. As mentioned, several potential applications with low cost and invasiveness can promote improvements mainly in prevention and patient follow-up.

 

SUPPLEMENTARY MATERIALS

Supplementary Material  

ACKNOWLEDGEMENTS

We thank Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for supporting the Programa de Bioinformática, Fundação da Universidade Federal do Paraná (FUNPAR), Fundação Araucária - Novo Arranjo de Pesquisa e Inovação (NAPI) Bioinformática (grant PDI-66/2021), Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq; grant 440412/2022-6), and Universidade Federal do Paraná (UFPR) for funding the equipment.

 

REFERENCES

1. Lima ES, Perico CP, Nichio BTL, Silveira RD, Raittz RT, Guizelini D, et al. Relationship between emotion and health: a two-way street. Braz J Dev. 2023;9:819-12.
2. Luneski A, Konstantinidis E, Bamidis PD. Affective medicine. A review of affective computing efforts in medical informatics. Methods Inf Med. 2010;49:207-18.
3. Pert CB. Molecules of emotion: Why you feel the way you feel. New York: Simon and Schuster; 1997.
4. Amkraut A, Solomon GF. From the symbolic stimulus to the pathophysiologic response: immune mechanisms. Int J Psychiatry Med. 1974;5:541-63.
5. Picard RW. Affective medicine: technology with emotional intelligence. Stud Health Technol Inform. 2002;80:69-83.
6. Dzedzickis A, Kaklauskas A, Bucinskas V. Human emotion recognition: review of sensors and methods. Sensors (Basel). 2020;20:592.
7. Sun L, Peräkylä J, Hartikainen KM. Frontal alpha asymmetry, a potential biomarker for the effect of neuromodulation on brain's affective circuitry-preliminary evidence from a deep brain stimulation study. Front Hum Neurosci. 2017;11:584.
8. Damásio AR. O erro de Descartes: emoção, razão e o cérebro humano. São Paulo: Companhia das Letras; 2012.
9. Pace-Schott EF, Amole MC, Aue T, Balconi M, Bylsma LM, Critchley H, et al. Physiological feelings. Neurosci Biobehav Rev. 2019;103:267-304.
10. D'Hondt F, Lassonde M, Collignon O, Dubarry AS, Robert M, Rigoulot S, et al. Early brain-body impact of emotional arousal. Front Hum Neurosci. 2010;4:33.
11. Cacioppo JT, Tassinary LG, Berntson GG. Handbook of psychophysiology. Cambridge: Cambridge University Press; 2016.
12. Picard R. Affective computing. Cambridge: The MIT Press; 1997.
13. Reynolds C, Picard R. Ethical evaluation of displays that adapt to affect. Cyberpsychol Behav. 2004;7:662-6.
14. Pinto G, Carvalho JM, Barros F, Soares SC, Pinho AJ, Brás S. Multimodal emotion evaluation: a physiological model for cost-effective emotion classification. Sensors (Basel). 2020;20:3510.
15. Russell JA. Core affect and the psychological construction of emotion. Psychol Rev. 2003;110:145-72.
16. Mauss IB, Robinson MD. Measures of emotion: A review. Cogn Emot. 2009;23:209-237.
17. Sreeja OPS, Mahalakshmi GS. Emotion models: a review. Int J Control Theory Appl. 2017;10:651-7.
18. Harmon-Jones E, Harmon-Jones C, Summerell E. On the importance of both dimensional and discrete models of emotion. Behav Sci (Basel). 2017;7:66.
19. Lange J, Dalege J, Borsboom D, van Kleef GA, Fischer AH. Toward an integrative psychometric model of emotions. Perspect Psychol Sci. 2020;15:444-68.
20. Wang Z, Ho S-B, Cambria E. A review of emotion sensing: categorization models and algorithms. Multimed Tools Appl. 2020;79:553-35.
21. Ekman P. Are there basic emotions? Psychol Rev. 1992;99:550-3.
22. Sprengelmeyer R, Rausch M, Eysel UT, Przuntek H. Neural structures associated with recognition of facial expressions of basic emotions. Proc Biol Sci. 1998;265:1927-31.
23. Rukavina S, Gruss S, Hoffmann H, Tan JW, Walter S, Traue HC. Affective computing and the impact of gender and age. PLoS One. 2016;11:e0150584.
24. Hasnul MA, Aziz NAA, Alelyani S, Mohana M, Aziz AA. Electrocardiogram-based emotion recognition systems and their applications in healthcare - A review. Sensors (Basel). 2021;21:5015.
25. Mehmood RM, Lee HJ. Emotion classification of EEG brain signal using SVM and KNN. In: 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). Turin: IEEE; 2015. p. 1-5.
26. Domínguez-Jiménez JA, Campo-Landines KC, Martínez-Santos JC, Delahoz EJ, Contreras-Ortiz SH. A machine learning model for emotion recognition from physiological signals. Biomed Signal Process Control. 2020;55:101646.
27. Goldstein P, Ashar Y, Tesarz J, Kazgan M, Cetin B, Wager TD. Emerging clinical technology: application of machine learning to chronic pain assessments based on emotional body maps. Neurotherapeutics. 2020;17:774-83.
28. Liu Z, Li C, Gao X, Wang G, Yang J. Ensemble-based depression detection in speech. In: 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Kansas City: IEEE; 2017. p. 975-80.
29. Dham S, Sharma A, Dhall A. Depression scale recognition from audio, visual and text analysis. arXiv. 2017 Sep 18;709.05865. doi: 10.48550/arXiv.1709.05865. Preprint
30. He L, Cao C. Automated depression analysis using convolutional neural networks from speech. J Biomed Inform. 2018;83:103-11.
31. Dar MN, Akram MU, Yuvaraj R, Khawaja SG, Murugappan M. EEG-based emotion charting for Parkinson's disease patients using Convolutional Recurrent Neural Networks and cross dataset learning. Comput Biol Med. 2022;144:105327.
32. Guo W, Yang H, Liu Z, Xu Y, Hu B. Deep Neural Networks for depression recognition based on 2D and 3D facial expressions under emotional stimulus tasks. Front Neurosci. 2021;15:609760.
33. Hansen L, Zhang YP, Wolf D, Sechidis K, Ladegaard N, Fusaroli R. A generalizable speech emotion recognition model reveals depression and remission. Acta Psychiatr Scand. 2022;145:186-99.
34. Mollahosseini A, Hasani B, Mahoor MH. AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput. 2019;10:18-31.
35. Busso C, Bulut M, Lee C-C, Kazemzadeh A, Mower E, Kim S, et al. IEMOCAP: interactive emotional dyadic motion capture database. Lang Resour Eval. 2008;42:335-59.
36. Peng RD. Reproducible research and Biostatistics. Biostatistics. 2009;10:405-8.
37. Ekman P, Friesen WV. Facial action coding system. Environ Psychol Nonverbal Behav. 1978.
38. Han J, Zhang Z, Ren Z, Schuller B. Implicit fusion by joint audiovisual training for emotion recognition in mono modality; In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton: IEEE; 2019. p. 5861-5.
39. Pelachaud C. Modelling multimodal expression of emotion in a virtual agent. Philos Trans R Soc Lond B Biol Sci. 2009;364:3539-48.
40. Pourhoseingholi MA, Vahedi M, Rahimzadeh M. Sample size calculation in medical studies. Gastroenterol Hepatol Bed Bench. 2013;6:14-7.
41. Peng RD. Reproducible research in computational science. Science. 2011;334:1226-7.
42. Rincon JA, Costa A, Novais P, Julian V, Carrascosa C. Intelligent wristbands for the automatic detection of emotional states for the elderly. In: Intelligent Data Engineering and Automated Learning - IDEAL 2018: 19th International Conference. Madrid: Springer; 2018, p. 520-30.
43. Hui TKL, Sherratt RS. Coverage of emotion recognition for common wearable biosensors. Biosensors (Basel). 2018;8:30.
44. Ménard M, Richard P, Hamdi H, Daucé B, Yamaguchi T [Internet]. Emotion recognition based on heart rate and skin conductance. 2015 [cited 2025 Apr 16]. https://www.scitepress.org/papers/2015/52411/52411.pdf
45. Bos DO. EEG-based emotion recognition the influence of visual and auditory stimuli [Thesis]. Enschedue: University of Twente; 2006.
46. Nagy SA, Kürtös Z, Németh N, Perlaki G, Csernela E, Lakner FE, et al. Childhood maltreatment results in altered deactivation of reward processing circuits in depressed patients: A functional magnetic resonance imaging study of a facial emotion recognition task. Neurobiol Stress. 2021;15:100399.
47. Leng H, Lin Y, Zanzi L. An experimental study on physiological parameters toward driver emotion recognition. In: Dainoff MJ, editor. Ergonomics and Health Aspects of Work with Computers. Berlin, Heidelberg: Springer; 2007. p. 237-46.
48. Pantic M, Bartlett MS. Machine analysis of facial expressions. In:Delac K, Grgic M, editors. Face recognition. Vienna: I-Tech Education and Publishing; 2007. p. 377-416.
49. Li S, Deng W. Deep facial expression recognition: a survey. IEEE Trans Affect Comput. 2022;13:1195-215.
50. He Z, Zhong Y, Pan J. An adversarial discriminative temporal convolutional network for EEG-based cross-domain emotion recognition. Comput Biol Med. 2022;141:105048.
51. Sperl MFJ, Panitz C, Rosso IM, Dillon DG, Kumar P, Hermann A, et al. Fear extinction recall modulates human frontomedial theta and amygdala activity. Cereb Cortex. 2019;29:701-15.
52. LeDoux JE. Brain mechanisms of emotion and emotional learning. Curr Opin Neurobiol. 1992;2:191-7.
53. Blood AJ, Zatorre RJ, Bermudez P, Evans AC. Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions. Nature Neurosci. 1999;2:382-7.
54. Hamann S, Mao H. Positive and negative emotional verbal stimuli elicit activity in the left amygdala. Neuroreport. 2002;13:15-9.
55. Zhang Z, Zhong S-H, Liu Y. GANSER: a self-supervised data augmentation framework for eeg-based emotion recognition. arXiv. 2021 Sep 7;2109.03124. doi: 10.48550/arXiv.2109.03124. Preprint
56. Ozdemir MA, Degirmenci M, Izci E, Akan A. EEG-based emotion recognition with deep convolutional neural networks. Biomed Tech (Berl). 2020:/j/bmte.ahead-of-print/bmt-2019-0306/bmt-2019-0306.xml.
57. Liu H, Zhang Y, Li Y, Kong X. Review on emotion recognition based on electroencephalography. Front Comput Neurosci. 2021;15:758212.
58. Rahman MM, Sarkar AK, Hossain MA, Hossain MS, Islam MR, Hossain MB, et al. Recognition of human emotions using EEG signals: a review. Comput Biol Med. 2021;136:104696.
59. Pane ES, Hendrawan MA, Wibawa AD, Purnomo MH. Identifying rules for electroencephalograph (EEG) emotion recognition and classification. In: 5th International Conference on Instrumentation, Communications, Information Technology, and Biomedical Engineering (ICICI-BME). Bandung: IEEE, 2017. p. 167-72.
60. Chang H, Zong Y, Zheng W, Tang C, Zhu J, Li X. Depression assessment method: an EEG emotion recognition framework based on spatiotemporal neural network. Front Psychiatry. 2022;12:837149.
61. Li M, Lu B-L. Emotion classification based on gamma-band EEG. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Minneapolis: IEEE; 2009. p. 1223-6.
62. Park M-S, Oh H-S, Jeong H, Sohn J-H. EEG-based emotion recognition during emotionally evocative films. In: International Winter Workshop on Brain-Computer Interface (BCI). Gangwon: IEEE; 2013. p. 56-7.
63. Pan J, Yang F, Qiu L, Huang H. Fusion of EEG-based activation, spatial, and connection patterns for fear emotion recognition. Comput Intell Neurosci. 2022;2022:3854513.
64. Rainville P, Bechara A, Naqvi N, Damasio AR. Basic emotions are associated with distinct patterns of cardiorespiratory activity. Int J Psychophysiol. 2006;61:5-18.
65. Lang PJ, Greenwald MK, Bradley MM, Hamm AO. Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology. 1993;30:261-73.
66. Klintworth A, Ajtay Z, Paljunite A, Szabados S, Hejjel L. Heart rate asymmetry follows the inspiration/expiration ratio in healthy volunteers. Physiol Meas. 2012;33:1717-31.
67. Fredrickson BL. Cultivating positive emotions to optimize health and well-being. Prev Treat. 2000;3:1.
68. Kubzansky LD, Kawachi I. Going to the heart of the matter: do negative emotions cause coronary heart disease? J Psychosom Res. 2000;48:323-37.
69. Clay-Warner J, Robinson DT. Infrared thermography as a measure of emotion response. Emotion Rev. 2015;7:157-62.
70. Cho Y, Julier SJ, Bianchi-Berthouze N. Instant stress: detection of perceived mental stress through smartphone photoplethysmography and thermal imaging. JMIR Ment Health. 2019;6:e10140.
71. Marqués-Sánchez P, Liébana-Presa C, Benítez-Andrades JA, Gundín-Gallego R, Álvarez-Barrio L, Rodríguez-Gonzálvez P. Thermal infrared imaging to evaluate emotional competences in nursing students: a first approach through a case study. Sensors (Basel). 2020;20:2502.
72. Van Dooren M, de Vries JJ, Janssen JH. Emotional sweating across the body: comparing 16 different skin conductance measurement locations. Physiol Behav. 2012;106:298-304.
73. Bradley MM, Lang PJ. Emotion and motivation. In:Cacioppo JT, Tassinary LG, Berntson GG, editors. Handbook of psychophysiology. 3rd ed. Cambridge: Cambridge University Press; 2007.
74. Rosebrock LE, Hoxha D, Norris C, Cacioppo JT, Gollan JK. Skin conductance and subjective arousal in anxiety, depression, and comorbidity. J Psychophysiol. 2017;31:145-57.
75. Shu L, Yu Y, Chen W, Hua H, Li Q, Jin J, et al. Wearable emotion recognition using heart rate data from a smart bracelet. Sensors (Basel). 2020;20:718.
76. Zhang Z, Song Y, Cui L, Liu X, Zhu T. Emotion recognition based on customized smart bracelet with built-in accelerometer. PeerJ. 2016;4:e2258.
77. Liu W, Liu Z. User experience evaluation of intelligent sports bracelet based on multi-factor fusion. Int J Prod Dev. 2022;26:102-16.
78. Verma GK, Tiwary US. Multimodal fusion framework: a multiresolution approach for emotion classification and recognition from physiological signals. Neuroimage. 2014;102 Pt 1:162-72.
79. Chaparro V, Gomez A, Salgado A, Quintero PL, Lopez N, Villa LF. Emotion recognition from EEG and facial expressions: a multimodal approach. In: 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu: IEEE; 2018. p. 530-3.
80. Flynn M, Effraimidis D, Angelopoulou A, Kapetanios E, Williams D, Hemanth J, et al. Assessing the effectiveness of automated emotion recognition in adults and children for clinical investigation. Front Hum Neurosci. 2020;14:70.
81. Bonanno G, Keltner D. Brief report: The coherence of emotion systems: comparing "on-line" measures of appraisal and facial expressions, and self-report. Cogn Emot. 2004;18:431-44.
82. Reisenzein R, Studtmann M, Horstmann G. Coherence between emotion and facial expression: evidence from laboratory experiments. Emot Rev. 2013;5:16-23.
83. Ekman P. What scientists who study emotion agree about. Perspect Psychol Sci. 2016;11:31-4.
84. Nemiah JC, Freyberger H, Sifneos PE. Alexithymia: a view of the psychosomatic process. Mod Trends Psychosom Med. 1976;3:430-9.
85. Kooiman CG, Spinhoven P, Trijsburg RW. The assessment of alexithymia: a critical review of the literature and a psychometric study of the Toronto Alexithymia Scale-20. J Psychosom Res. 2002;53:1083-90.
86. Trevisan DA, Bowering M, Birmingham E. Alexithymia, but not autism spectrum disorder, may be related to the production of emotional facial expressions. Mol Autism. 2016;7:46.
87. Olugbade TA, Aung MSH, Bianchi-Berthouze N, Marquardt N, Williams AC. Bi-modal detection of painful reaching for chronic pain rehabilitation systems. In: Proceedings of the 16th International Conference on Multimodal Interaction. New York: Association for Computing Machinery; 2014. p. 455-8.
88. Kappesser J, Williams AC. Pain and negative emotions in the face: judgements by health care professionals. Pain. 2002;99:197-206.
89. Kunz M, Seuss D, Hassan T, Garbas JU, Siebers M, Schmid U, et al. Problems of video-based pain detection in patients with dementia: a road map to an interdisciplinary solution. BMC Geriatr. 2017;17:33.
90. Herr K, Bjoro K, Decker S. Tools for assessment of pain in nonverbal older adults with dementia: a state-of-the-science review. J Pain Symptom Manage. 2006;31:170-92.
91. Aung MSH, Kaltwang S, Romera-Paredes B, Martinez B, Singh A, Cella M, et al. The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal EmoPain dataset. IEEE Trans Affect Comput. 2016;7:435-51.
92. Birnie KA, Hundert AS, Lalloo C, Nguyen C, Stinson JN. Recommendations for selection of self-report pain intensity measures in children and adolescents: a systematic review and quality assessment of measurement properties. Pain. 2019;160:5-18.
93. Matsangidou M, Liampas A, Pittara M, Pattichi CS, Zis P. Machine learning in pain medicine: an up-to-date systematic review. Pain Ther. 2021;10:1067-84.
94. Huang H, Xie Q, Pan J, He Y, Wen Z, Yu R, et al. An EEG-based brain computer interface for emotion recognition and its application in patients with disorder of consciousness. IEEE Trans Affect Comput. 2019;12:832-42.
95. He Z, Li Z, Yang F, Wang L, Li J, Zhou C, et al. Advances in multimodal emotion recognition based on brain-computer interfaces. Brain Sci. 2020;10:687.
96. Pan J, Xie Q, Huang H, He Y, Sun Y, Yu R, et al. Emotion-related consciousness detection in patients with disorders of consciousness through an EEG-based BCI system. Front Hum Neurosci. 2018;12:198.
97. Harrison AH, Connolly JF. Finding a way in: a review and practical evaluation of fMRI and EEG for detection and assessment in disorders of consciousness. Neurosci Biobehav Rev. 2013;37:1403-19.
98. Gavrilova ML, Ahmed F, Bari AH, Liu R, Liu T, Maret Y, et al. Multi-modal motion-capture-based biometric systems for emergency response and patient rehabilitation. In: Kisku DK, Gupta P, Sing JK, editors. Design and implementation of healthcare biometric systems. Hershey: IGI global, 2021. p. 653-78.
99. Shany-Ur T, Rankin KP. Personality and social cognition in neurodegenerative disease. Curr Opin Neurol. 2011;24:550-5.
100. Karve J, Mendez MF, Kaiser N, Jimenez E, Mather M, Shapira JS. Skin conductance levels may reflect emotional blunting in behavioral variant frontotemporal dementia. J Neuropsychiatry Clin Neurosci. 2014;26:227-32.
101. Najström M, Jansson B. Skin conductance responses as predictor of emotional responses to stressful life events. Behav Res Ther. 2007;45:2456-63.
102. Sorinas J, Ferrández JM, Fernández E. Perspectives on affective computing for the early diagnosis of neurodegenerative diseases. In: 3rd Workshop on Affective Computing and Context Awareness in Ambient Intelligence. Cartagena: AfCAI; 2019.
103. Smith E, Storch EA, Vahia I, Wong STC, Lavretsky H, Cummings JL, et al. Affective computing for late-life mood and cognitive disorders. Front Psychiatry. 2021;12:782183.
104. Pontieri FE, Assogna F, Stefani A, Pierantozzi M, Meco G, Benincasa D, et al. Sad and happy facial emotion recognition impairment in progressive supranuclear palsy in comparison with Parkinson's disease. Parkinsonism Relat Disord. 2012;18:871-5.
105. Wang L, Hao J, Zhou TH. ECG multi-emotion recognition based on heart rate variability signal features mining. Sensors. 2023;23:8636.
106. Pellissier S, Bonaz B. The Place of stress and emotions in the irritable bowel syndrome. Vitam Horm. 2017;103:327-54.
107. Grünerbl A, Muaremi A, Osmani V, Bahle G, Ohler S, Tröster G, et al. Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE J Biomed Health Inform. 2015;19:140-8.
108. Prociow P, Wac K, Crowe J. Mobile psychiatry: towards improving the care for bipolar disorder. Int J Ment Health Syst. 2012;6:5.
109. Cosic K, Popovic S, Horvat M, Kukolja D, Dropuljic B, Kovac B, et al. Computer-aided psychotherapy based on multimodal elicitation, estimation and regulation of emotion. Psychiatr Danub. 2013;25:340-6.
110. Berens S, Schaefert R, Ehrenthal JC, Baumeister D, Eich W, Tesarz J. Different dimensions of affective processing in patients with irritable bowel syndrome: a multi-center cross-sectional study. Front Psychol. 2021;12:625381.
111. Ke J, Qi R, Liu C, Xu Q, Wang F, Zhang L, et al. Abnormal regional homogeneity in patients with irritable bowel syndrome: a resting-state functional MRI study. Neurogastroenterol Motil. 2015;27:1796-803.
112. Lee LN, Kim MJ. A critical review of smart residential environments for older adults with a focus on pleasurable experience. Front Psychol. 2020;10:3080.
113. Jeon M, Walker BN. What to detect? Analyzing factor structures of affect in driving contexts for an emotion detection and regulation system. Pro. Hum Factors Ergon Soc Annu Meet. 2011;55:1889-93.
114. Ji Q, Zhu Z, Lan P. Real-time nonintrusive monitoring and prediction of driver fatigue. IEEE Trans Veh Technol. 2004;53:1052-68.
115. Yin J-L, Chen B-H, Lai K-H R, Li Y. Automatic dangerous driving intensity analysis for advanced driver assistance systems from multimodal driving signals. IEEE Sens J. 2017;18:4785-94.
116. Arshad S, Feng C, Elujide I, Zhou S, Liu Y. SafeDrive-Fi: A multimodal and device free dangerous driving recognition system using WiFi. In: 2018 IEEE International Conference on Communications (ICC). Kansas City: IEEE; 2018. p. 1-6.
117. M'bailara K, Atzeni T, Contrand B, Derguy C, Bouvard MP, Lagarde E, et al. Emotional reactivity: Beware its involvement in traffic accidents. Psychiatry Res. 2018;262:290-4.
118. Tzirakis P, Zafeiriou S, Schuller B. Chapter 18 - Real-world automatic continuous affect recognition from audiovisual signals. In:Alameda-Pineda X, Ricci E, Sebe N, editors. Multimodal Behavior Analysis in the Wild: Advances and Challenges. San Diego: Elsevier; 2019. p. 387-406.
119. Kaya H, Gürpınar F, Salah AA. Video-based emotion recognition in the wild using deep transfer learning and score fusion. Image Vis Comput. 2017;65:66-75.
120. Samadiani N, Huang G, Luo W, Chi C-H, Shu Y, Wang R, et al. A multiple feature fusion framework for video emotion recognition in the wild. Concurr Comput Pract Exp. 2022;34:e5764.
121. Mitenkova A, Kossaifi J, Panagakis Y, Pantic M. Valence and arousal estimation in-the-wild with tensor methods. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). Lille: IEEE; 2019. p. 1-7.
122. Liu H, Zeng J, Shan S. Facial expression recognition for in-the-wild videos. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020). Buenos Aires: IEEE; 2020. p. 615-8.
123. Kollias D, Zafeiriou S. Aff-Wild2: extending the Aff-Wild database for affect recognition. arXiv. 2019 Nov 11;1811.07770. doi: 10.48550/arXiv.1811.07770. Preprint
124. Lahasan B, Lutfi SL, San-Segundo R. A survey on techniques to handle face recognition challenges: occlusion, single sample per subject and expression. Artif Intell Rev. 2019;52:949-79.
125. Nojavanasghari B, Hughes CE, Baltrusaitis T, Morency LP. Hand2Face: automatic synthesis and recognition of hand over face occlusions. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). San Antonio: IEEE; 2017. p. 209-15.
126. Li Y, Zeng J, Shan S, andChen X. Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans Image Process. 2018;28:2439-50.
127. Dagnes N, Vezzetti E, Marcolin F, Tornincasa S. Occlusion detection and restoration techniques for 3D face recognition: a literature review. Mach Vis Appl. 2018;29:789-813.
128. Duan Q, Zhang L. Look more into occlusion: realistic face frontalization and recognition with BoostGAN. IEEE Trans Neural Netw Learn Syst. 2020;32:214-28.
129. Avila AR, Akhtar Z, Santos JF, O'Shaughnessy D, Falk TH. Feature pooling of modulation spectrum features for improved speech emotion recognition in the wild. IEEE Trans Affect Comput. 2018;12:177-88.
130. Kim DH, Lee MK, Choi DY, Song BC. Multi-modal emotion recognition using semi-supervised learning and multiple neural networks in the wild. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction. New York: AMC; 2017. p. 529-35.
131. Chen S, Jin Q, Zhao J, Wang S. Multimodal multi-task learning for dimensional and continuous emotion recognition. In: Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge. New York: AMC; 2017. p. 19-26.
132. Kanjo E, Younis EM, Ang CS. Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection. Inf Fusion. 2019;49:46-56.
133. Mittal T, Bhattacharya U, Chandra R, Bera A, Manocha D. M3er: multiplicative multimodal emotion recognition using facial, textual, and speech cues. In: Proc AAAI Conf Artif Intell. 2020;34:1359-67.
134. Wang X-W, Nie D, Lu B-L. EEG-based emotion recognition using frequency domain features and support vector machines. In: Neural Information Processing, ICONIP 2011. Berlin: Springer; 2011. p. 734-43.
135. Lin YP, Wang CH, Wu TL, Jeng SK, Chen JH. Multilayer perceptron for EEG signal classification during listening to emotional music. In: TENCON 2007 - 2007 IEEE Region 10 Conference. Taipei: IEEE; 2007. p. 1-3.
136. Pan J, Li Y, Wang J. An EEG-based brain-computer interface for emotion recognition. In: 2016 International Joint Conference on Neural Networks (IJCNN). Vancouver: IEEE; 2016. p. 2063-7.
137. Yu SN, Chen SF. Emotion state identification based on heart rate variability and genetic algorithm. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Milan: IEEE; 2015. p. 538-41.
138. Boccanfuso L, Wang Q, Leite I, Li B, Torres C, Chen L, et al. A thermal emotion classifier for improved human-robot interaction. In: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). New York: IEEE; 2016. p. 718-23.
139. Wei Y, Wu Y, Tudor J. A real-time wearable emotion detection headband based on EEG measurement. Sens Actuators A Phys. 2017;263:614-21.
140. Jerritta S, Murugappan M, Khairunizam W, Yaacob S. Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform. Expert Syst. 2014;31:110-20.
141. Alsolamy M, Fattouh A. Emotion estimation from EEG signals during listening to Quran using PSD features. In: 2016 7th International Conference on Computer Science and Information Technology (CSIT). Amman: IEEE; 2016. p. 1-5.
142. Petrantonakis PC, Hadjileontiadis LJ. Emotion recognition from EEG using higher order crossings. IEEE Trans Inf Technol Biomed. 2009;14:186-97.
143. Goulart C, Valadão C, Delisle-Rodriguez D, Funayama D, Favarato A, Baldo G, et al. Visual and thermal image processing for facial specific landmark detection to infer emotions in a child-robot interaction. Sensors. 2019;19:2844.
144. Jalilifard A, Pizzolato EB, Islam MK. Emotion classification using single-channel scalp-EEG recording. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Orlando: IEEE; 2016. p. 845-9.
145. Shahabi H, Moghimi S. Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity. Comput Human Behav. 2016;58:231-9.
146. Zhang Z, Wang X, Li P, Chen X, Shao L. Research on emotion recognition based on ECG signal. J Phys Conf Ser. 2020;1678:012091.
147. Murugappan M, Murugappan S, Zheng BS. Frequency band analysis of electrocardiogram (ECG) signals for human emotional state classification using discrete wavelet transform (DWT). J Phys Ther Sci. 2013;25:753-9.
148. Taran S, Bajaj V. Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Comput Methods Programs Biomed. 2019;173:157-65.
149. Khare SK, Bajaj V. An evolutionary optimized variational mode decomposition for emotion recognition. IEEE Sens J. 2020;21:2035-42.
150. Mehmood RM, Du R, Lee HJ. Optimal feature selection and deep learning ensembles method for emotion recognition from human brain EEG sensors. IEEE Access. 2017;5:14797-806.
151. Mehmood RM, Lee HJ. A novel feature extraction method based on late positive potential for emotion recognition in human brain signal patterns. Comput Electr Eng. 2016;53:444-57.
152. Raheel A, Anwar SM, Majid M. Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography. Multimed Tools Appl. 2019;78:13971-85.
153. Dissanayake T, Rajapaksha Y, Ragel R, Nawinne I. An ensemble learning approach for electrocardiogram sensor based human emotion recognition. Sensors (Basel). 2019;19:4495.
154. Guo HW, Huang YS, Lin CH, Chien JC, Haraikawa K, Shieh JS. Heart rate variability signal features for emotion recognition by using principal component analysis and support vectors machine. In: 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE). Taichung: IEEE; 2016. p. 274-7.
155. Yu C, Tapus A. Multimodal emotion recognition with thermal and RGB-D cameras for human-robot interaction. In: 2020 ACM/IEEE International Conference on Human-Robot Interaction. New York: Association for Computing Machinery; 2020. p. 532-4.
156. Lin YP, Wang CH, Jung TP, Wu TL, Jeng SK, Duann JR, et al. EEG-based emotion recognition in music listening. IEEE Trans Biomed Eng. 2010;57:1798-806.
157. Basu A, Routray A, Shit S, Deb AK. Human emotion recognition from facial thermal image based on fused statistical feature and multi-class SVM. In: 2015 Annual IEEE India Conference (INDICON). New Delhi: IEEE; 2015. p. 1-5.
158. Bhatti AM, Majid M, Anwar SM, Khan B. Human emotion recognition and analysis in response to audio music using brain signals. Comput Human Behav. 2016;65:267-75.
159. Zhuang N, Zeng Y, Yang K, Zhang C, Tong L, Yan B. Investigating patterns for self-induced emotion recognition from EEG signals. Sensors. 2018;18:841.
160. Tarvainen MP, Koistinen AS, Valkonen-Korhonen M, Partanen J, Karjalainen PA. Analysis of galvanic skin responses with principal components and clustering techniques. IEEE Trans Biomed Eng. 2001;48:1071-9.
161. Kolli A, Fasih A, Al Machot F, Kyamakya K. Non-intrusive car driver's emotion recognition using thermal camera. In: Proceedings of the Joint INDS'11 & ISTET'11. Klagenfurt am Wörthersee: IEEE; 2011. p. 1-5.
162. Mikhail M, El-Ayat K, El Kaliouby R, Coan J, Allen JJ. Emotion detection using noisy EEG data. In: Proceedings of the 1st Augmented Human International Conference. New York: Association for Computing Machinery; 2010. p. 1-7.
163. Cruz-Albarran IA, Benitez-Rangel JP, Osornio-Rios RA, Morales-Hernandez LA. Human emotions detection based on a smart-thermal system of thermographic images. Infrared Phys Technol. 2017;81:250-61.
164. Quiroz JC, Yong MH, Geangu E. Emotion-recognition using smart watch accelerometer data: Preliminary findings. In: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. New York: Association for Computing Machinery; 2017. p. 805-12.
165. Cui L, Li S, Zhu T. Emotion detection from natural walking. In: Revised Selected Papers of the Second International Conference on Human Centered Computing. Berlin: Springer; 2016. p. 23-33.
166. Selvaraj J, Murugappan M, Wan K, Yaacob S. Classification of emotional states from electrocardiogram signals: A non-linear approach based on Hurst. Biomed Eng Online. 2013;12:1-18.
167. Nayak S, Nagesh B, Routray A, Sarma M. A human-computer interaction framework for emotion recognition through time-series thermal video sequences. Comput Electr Eng. 2021;93:107280.
168. Agrafioti F, Hatzinakos D, Anderson AK. ECG pattern analysis for emotion detection. IEEE Trans Affect Comput. 2011;3:102-15.
169. Xun L, Zheng G. ECG signal feature selection for emotion recognition. Telkomnika. 2013;11:1363-70.
170. Wang WH, Qiu YH, Li GY. Electrocardiography recording, feature extraction and classification for emotion recognition. In: 2009 WRI World Congress on Computer Science and Information Engineering. Los Angeles: IEEE; 2009. p. 168-72.
171. Guo X. Study of emotion recognition based on electrocardiogram and RBF neural network. Procedia Eng. 2011;15:2408-12.
172. Wu G, Liu G, Hao M. The analysis of emotion recognition from GSR based on PSO. In: 2010 International Symposium on Intelligence Information Processing and Trusted Computing. Huanggang: IEEE; 2010. p. 360-3.
173. Ferdinando H, Seppänen T, Alasaarela E. Bivariate empirical mode decomposition for ECG-based biometric identification with emotional data. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Jeju: IEEE; 2017. p. 450-3.
174. Chen S, Jiang K, Hu H, Kuang H, Yang J, Luo J, et al. Emotion recognition based on skin potential signals with a portable wireless device. Sensors. 2021;21(3):1018.
175. Nguyen H, Kotani K, Chen F, Le B. Estimation of human emotions using thermal facial information. In: Proceedings of the Fifth International Conference on Graphic and Image Processing. Hong Kong: SPIE; 2014. p. 126-30.
176. Ali M, Machot FA, Mosa AH, Kyamakya K. CNN-based subject-independent driver emotion recognition system involving physiological signals for ADAS. In:Schulze T, Müller B, Meyer G, eds. Advanced Microsystems for Automotive Applications 2016. Lecture Notes in Mobility. Cham: Springer; 2016. p. 125-38.
177. Wei W, Jia Q, Feng Y, Chen G. Emotion recognition based on weighted fusion strategy of multichannel physiological signals. Comput Intell Neurosci. 2018;2018:1-10.
178. Vijayan AE, Sen D, Sudheer A. EEG-based emotion recognition using statistical measures and auto-regressive modeling. In: 2015 IEEE International Conference on Computational Intelligence & Communication Technology (CICT). Ghaziabad: IEEE; 2015. p. 587-91.
179. Zamanian H, Farsi H. A new feature extraction method to improve emotion detection using EEG signals. Electron Lett Comput Vis Image Anal. 2018;17:29-44.
180. Chen G, Zhang X, Sun Y, Zhang J. Emotion feature analysis and recognition based on reconstructed EEG sources. IEEE Access. 2020;8:11907-16.
181. Jing C, Liu G, Hao M. The research on emotion recognition from ECG signal. In: 2009 International Conference on Information Technology and Computer Science. Kiev: IEEE; 2009. p. 497-500.
182. Xu Y, Liu GY. A method of emotion recognition based on ECG signal. In: 2009 International Conference on Computational Intelligence and Natural Computing. Wuhan: IEEE; 2009. p. 202-5.
183. Brás S, Ferreira JH, Soares SC, Pinho AJ. Biometric and emotion identification: An ECG compression based method. Front Psychol. 2018;9:467.
184. Alghowinem S, Goecke R, Wagner M, Alwabil A. Evaluating and validating emotion elicitation using English and Arabic movie clips on a Saudi sample. Sensors. 2019;19:2218.
185. Goulart C, Valadão C, Delisle-Rodriguez D, Caldeira E, Bastos T. Emotion analysis in children through facial emissivity of infrared thermal imaging. PLoS One. 2019;14:e0212928.
186. Balan O, Moise G, Moldoveanu A, Leordeanu M, Moldoveanu F. Fear level classification based on emotional dimensions and machine learning techniques. Sensors. 2019;19:1738.
187. Sarath S, Sarath S, Malavika Prasanna, Karthik. Human emotions recognition from thermal images using YOLO algorithm. In: 2020 International Conference on Communication and Signal Processing (ICCSP). Chennai: IEEE; 2020. p. 1139-42.
188. Batelaan NM, Seldenrijk A, van den Heuvel OA, van Balkom AJ, Kaiser A, Reneman L, et al. Anxiety, mental stress, and sudden cardiac arrest: epidemiology, possible mechanisms and future research. Front Psychiatry. 2022;12:2644.
189. Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, et al. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans Multimedia. 2010;12:682-91.
190. Huang Y, Zhu C, Feng Y, Ji Y, Song J, Wang K, et al. Comparison of three machine learning models to predict suicidal ideation and depression among Chinese adolescents: a cross-sectional study. J Affect Disord. 2022;319:221-8.
191. He Z, Zhong Y, Pan J. Emotion-related awareness detection for patients with disorders of consciousness via graph isomorphic network. In: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). Prague: IEEE; 2022. p. 3158-64.
192. Liu Y, Jiang C. Recognition of shooter's emotions under stress based on affective computing. IEEE Access. 2019;7:62338-43.
193. Varotto G, Fazio P, Sebastiano DR, Avanzini G, Franceschetti S, Panzica F, et al. Music and emotion: an EEG connectivity study in patients with disorders of consciousness. Annu Int Conf IEEE Eng Med Biol Soc. 2012;2012:5206-9.
194. Perez-Toro PA, Vasquez-Correa JC, Bocklet T, Nöth E, Orozco-Arroyave JR. User state modeling based on the arousal-valence plane: Applications in customer satisfaction and health-care. IEEE Trans Affect Comput. 2021;14:1533-46.
195. Pan J, Wang L, Huang H, Xiao J, Wang F, Liang Q, et al. A hybrid brain-computer interface combining P300 potentials and emotion patterns for detecting awareness in patients with disorders of consciousness. IEEE Trans Cogn Dev Syst. 2022;15:1386-95.
196. Hosseini SA, Khalilzadeh MA, Naghibi-Sistani MB, Homam SM. Emotional stress recognition using a new fusion link between electroencephalogram and peripheral signals. Iran J Neurol. 2015;14:142.
197. Moon SM, Qu Y. A quantitative approach for determining pilot affective patterns during soaring flight simulation. In: 2017 3rd IEEE International Conference on Computer and Communications (ICCC). Chengdu: IEEE; 2017. p. 2617-24.
198. Zolyomi A, Snyder J. Social-emotional-sensory design map for affective computing informed by neurodivergent experiences. In: Proceedings of the ACM on Human-Computer Interaction. New York: Association for Computing Machinery; 2021. p. 1-37.
199. Prossinger H, Hladký T, Boschetti S, Ríha D, Binter J. Determination of "neutral"-"pain", "neutral"-"pleasure", and "pleasure"-"pain" affective state distances by using AI image analysis of facial expressions. Technologies. 2022;10:75.
200. Bourvis N, Aouidad A, Spodenkiewicz M, Palestra G, Aigrain J, Baptista A, et al. Adolescents with borderline personality disorder show a higher response to stress but a lack of self-perception: Evidence through affective computing. Prog Neuropsychopharmacol Biol Psychiatry. 2021;111:110095.
201. ArulDass SD, Jayagopal P. Identifying complex emotions in alexithymia affected adolescents using machine learning techniques. Diagnostics. 2022;12:3188.
202. Akbulut FP, Ikitimur B, Akan A. Wearable sensor-based evaluation of psychosocial stress in patients with metabolic syndrome. Artif Intell Med. 2020;104:101824.
203. Sparrow L, Six H, Varona L, Janin O. Validation of affect-tag affective and cognitive indicators. Front Neuroinform. 2021;15:535542.
204. Filippou V, Theodosiou N, Nicolaou M, Constantinou E, Panayiotou G, Theodorou M. A wavelet-based approach for multimodal prediction of alexithymia from physiological signals. In: Companion Publication of the 2022 International Conference on Multimodal Interaction. New York: Association for Computing Machinery; 2022. p. 177-84.
205. Moretta P, Spisto M, Ausiello FP, Iodice R, De Lucia N, Santangelo G, et al. Alteration of interoceptive sensitivity: Expanding the spectrum of behavioural disorders in amyotrophic lateral sclerosis. Neurol Sci. 2022;43:5403-10.
206. Atee M, Hoti K, Hughes JD. A technical note on the PainChekT system: A web portal and mobile medical device for assessing pain in people with dementia. Front Aging Neurosci. 2018;10:117.
207. Kang S, Choi W, Park CY, Cha N, Kim A, Khandoker AH, et al. K-EmoPhone: a mobile and wearable dataset with in-situ emotion, stress, and attention labels. Sci Data. 2023;10:351.
208. Garcia-Cordero I, Migeot J, Fittipaldi S, Aquino A, Campo CG, García A, et al. Metacognition of emotion recognition across neurodegenerative diseases. Cortex. 2021;137:93-107.
209. Wu C, Sun J, Wang T, Zhao C, Zheng S, Lei C, et al. An application of affective computing on mental disorders: A resting state fNIRS study. IFAC-Pap Online. 2020;53:464-9.
210. Campbell E, Phinyomark A, Scheme E. Feature extraction and selection for pain recognition using peripheral physiological signals. Front Neurosci. 2019;13:437.
211. Wang T, Li C, Wu C, Zhao C, Sun J, Peng H, et al. A gait assessment framework for depression detection using Kinect sensors. IEEE Sensors J. 2020;21:3260-70.
212. Birba A, Fittipaldi S, Escobar JCC, Campo CG, Legaz A, Galiani A, et al. Multimodal neurocognitive markers of naturalistic discourse typify diverse neurodegenerative diseases. Cereb Cortex. 2022;32:3377-91.
213. Caldo D, Bologna S, Conte L, Amin MS, Anselma L, Basile V, et al. Machine learning algorithms distinguish discrete digital emotional fingerprints for web pages related to back pain. Sci Rep. 2023;13:4654.
214. Garg D, Verma GK, Singh AK. A review of deep learning based methods for affect analysis using physiological signals. Multimed Tools Appl. 2023:82:26089-134.
215. Majumdar S, Al-Habaibeh A, Omurtag A, Shakmak B, Asrar M. A novel approach for communicating with patients suffering from completely locked-in-syndrome (CLIS) via thoughts: Brain computer interface system using EEG signals and artificial intelligence. Neurosci Inform. 2023;3:100126.
216. Bhachawat S, Shriram E, Srinivasan K, Hu YC. Leveraging computational intelligence techniques for diagnosing degenerative nerve diseases: a comprehensive review, open challenges, and future research directions. Diagnostics. 2023;13:288.
217. Sigcha L, Borzì L, Amato F, Rechichi I, Ramos-Romero C, Cárdenas A, et al. Deep learning and wearable sensors for the diagnosis and monitoring of Parkinson's disease: a systematic review. Expert Syst Appl. 2023;229:120541.
218. Nour M, Senturk U, Polat K. Diagnosis and classification of Parkinson's disease using ensemble learning and 1D-PDCOVNN. Comput Biol Med. 2023;161:107031.
219. Weber I, Oehrn CR. NOLITIA: an open-source toolbox for non-linear time series analysis. Front Neuroinform. 2022;16:876012.
220. Shabanpour M, Kaboodvand N, Iravani B. Parkinson's disease is characterized by sub-second resting-state spatio-oscillatory patterns: a contribution from deep convolutional neural network. Neuroimage Clin. 2022;36:103266.
221. Sarafraz G, Behnamnia A, Hosseinzadeh M, Balapour A, Meghrazi A, Rabiee HR. Domain adaptation and generalization on functional medical images: a systematic survey. arXiv. 2022 Dec 4;2212.03176. doi: 10.48550/arXiv.2212.03176. Preprint
222. R Sharma R, Meena HK. EMHM: A novel hybrid model for the emotion recognition based on EEG signals. In: 2023 19th IEEE International Colloquium on Signal Processing & Its Applications (CSPA). Kedah: IEEE; 2023. p. 75-80.
223. Consedine NS, Moskowitz JT. The role of discrete emotions in health outcomes: A critical review. Appl Prev Psychol. 2007;12:59-75.
224. Gumpili SP, Das AV. Sample size and its evolution in research. IHOPE J Ophthalmol. 2022;1:9-13.
225. Czajkowski M, Kretowski M. Decision tree underfitting in mining of gene expression data. An evolutionary multi-test tree approach. Expert Syst Appl. 2019;137:392-404.
226. Zhang H, Zhang L, Jiang Y. Overfitting and underfitting analysis for deep learning based end-to-end communication systems. In: 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP). Xi'an: IEEE; 2019. p. 1-6.
227. Afzal S, Khan HA, Piran MJ, Lee JW. A comprehensive survey on affective computing; challenges, trends, applications, and future directions. IEEE Access. 2024;12:96150-68.
228. Benitez-Garcia G, Nakamura T, Kaneko M. Multicultural facial expression recognition based on differences of Western-Caucasian and East-Asian facial expressions of emotions. IEICE Trans Inf Syst. 2018;101:1317-24.
229. Khanh TLB, Kim SH, Lee G, Yang HJ, Baek ET. Korean video dataset for emotion recognition in the wild. Multimed Tools Appl. 2021;80:9479-92.
230. Lindquist KA, Siegel EH, Quigley KS, Barrett LF. The hundred-year emotion war: are emotions natural kinds or psychological constructions? Comment on Lench, Flores, and Bench (2011). Psychol Bull. 2013;139:255-63.
231. Krejcie RV. Determining sample size for research activities. Educ Psychol Meas. 1970;30:607-10.
232. Picard RW. Affective computing: Challenges. Int J Hum Comput Stud. 2003;59:55-64.

© 2026 All rights reserved