Multisensed Emotions as Adaptation Controllers in Human-to-Serious NeuroGames Communication

Emotion is defined (from Britannica) as a complex combination of experiences of consciousness, bodily sensation and behavior, reflecting personal significance of externally and/or internally triggered changes. As an indispensable part of human communication, emotions could be represented within the digital world as a means that could control human interaction with digital artifacts. When this is combined with gamification, a more engaging communication for the player could be established. In this article, we explore how multisensed emotions of players could be used as adaptation controllers within the emotion-adpting serious neurogames (SNGs). We first briefly describe the emotion sensing and recognition landscape, examining the emotion-source sensing and emotion inferring approaches. We then discuss the SNGs design framework and the way human-to-SNG communication is set. Additionally, we describe the way the adaptation of the SNG characteristics can be driven by the inferred emotions, towards the achievement of the desired SNG goals. In addition, we demonstrate the feasibility and design characteristics of multisensed emotion-adapting SNGs, by presenting a related application case, namely EmoSense, and evaluate its acceptance as a technological solution in the field of neurofeedback. Finally, we discuss challenges and opportunities in the field.


IntroductIon
Emotions are felt as time-dynamic experiences that reflect personally relevant changes in our inner and/or outer environment.These changes, perceived as either threats or opportunities to our comfort-zone that trigger affective experiences, help us to effectively cope with such fluctuations [1].Daily emotional life and social communication are influenced by emotional triggers and their flux-patterns across time can provide unique information for the status of our psychological well-being and used to understand, and even explain, differences across individuals with mental health or psychopathology [2].
Humans perceive emotions of others, fostering empathy, communication and emotion sharing.However, although this seems quite a natural, even spontaneous, human reaction, the similar task of recognizing emotions is a complex, power-intensive and computationally demanding task for machines [3].Nevertheless, the creation of empathic computing has become a central challenge during the last decade, attracting a great deal of attention from research, clinical and industry sectors.Applications in this sector have exponentially grown in parallel with the technological advances in capacity, performance, and intelligence of smart devices (e.g., smartphones, smartwatches, IoTs), along with the evolution of new AI-based emotion classification/prediction algorithms, emotion-related Web services, and new ways of relaying emotions during interactions on social networks, such as mixed/ virtual reality/metaverse [4].
Emotion-related data can be acquired from various sources of affective information, such as body reactions (e.g., facial expressions, gestures), changes in physiological signals (e.g., brainwaves, cortical hemodynamic activity, heart rate, respiration rate, temperature, electrodermal activity (EDA)), and means of communication and interaction (e.g., text/emoticons, images, video and sound).The acquisition of such data requires appropriate sensors that can focus on one (monomodal sensing) or a fusion of many (multimodal sensing) sources, providing rich, yet coded, information about the physiological responses and overt behavior, including facial expression or voice characteristics to emotional stimuli.To decode such emotion information, computational systems, available either as part of smart devices, and/or at the edge/cloud, communicating with the sensing systems and using emotional intelligence, are needed to provide emotion recognition outputs [5].
Well-known triggers of feelings, expressions, and physiological responses, which their transitory synchronization establishes the current emotional state, are games.In fact, game characteristics (e.g., story and characters, game mechanics and gameplay) can be designed, in order to maximize emotion elicitation during game play and allow different options for evaluating of the player's emotional and mental state [6].In this vein, Serious NeuroGames (SNGs) have been recently developed, where this nascent form of gaming involves brain-computer interfaces (BCI), such as electroencephalogram (EEG) technology, instead of traditional controllers.Despite the limited number of SNGs with limited number of game actions currently available in the market, SNGs have been developed for a variety of purposes, such as to improve learning and concentration skills, help people with Alzheimer's or Parkinson's Disease, Attention Deficit Hyperactivity Disorder (ADHD), Post-Traumatic Stress Disorder (PTSD), treat problems of chronic pain and depression, and facilitate stroke rehabilitation [7].Examples of SNGs include the recent MindMedia BrainAsssistant, Zukor Sports 1 Game Suite, along with some older ones, e.g., Syncself 2, AmbuRun, Neu-roRacer, NeuroMage.
In this article, we aim to give the readers an overview of using multisensed emotions as adaptation controllers in human-to-SNGs communication, spanning from the state-of-the-art to emerging approaches in this field.We propose a novel approach in the way human-to-SNG communication is set and we describe the way adaptation of the SNG characteristics can be driven by the inferred emotions, towards achieving the desired characterizing SNG goals.The main contributions of this article include: a) placement of SNGs in the communication path of the human-machine interaction, b) introduction of emotion-adapting design framework of SNGs, c) integration of multisensed biosignals sensing and analysis that trigger emotion-adapting feedback, and d) exploration of emotion-based adaptation schemes of SNGs.To further exemplify the feasibility and design characteristics of such multisensed emotion-adapted SNGs, we present a related application case, namely Emo-Sense that highlights the uptake propensity of SNGs in the wider community.
In the remainder of this article, we first briefly describe the emotion sensing and recognition landscape, examining the emotion-source sensing and emotion inferring approaches.Then, we discuss the SNGs design framework and the way human-to-SNG communication is set.Next, we describe the way adaptation of the SNG characteristics can be driven by the inferred emotions, towards achieving the desired characterizing SNG goals.Then, we present EmoSense and evaluate its acceptance as a technological solution in the field of neurofeedback.Finally, we discuss the challenges and opportunities in this area, followed by the main conclusions.

EmotIon sEnsIng And rEcognItIon LAndscApE EmotIon sEnsIng
The human body is the locus of affective information; hence, emotion-related data can span from behavioral responses (e.g., facial expressions, social interaction) to physiological reactions (e.g., body signals alterations) [3].Considering the physical sensors embedded within mobile devices for sensing the external environment, their use as a data collection platform for unobtrusive sensing of emotion is natural.For example, smartphone keystroke dynamics provide digital biomarkers for detecting depressive tendencies [8].In addition, wearable devices, either commercial or customized (e.g., smartwatches, wristbands, smart rings, headset sensors) can acquire physiological signals, such as EEG, electrocardiogram (ECG), electromyogram (EMG), galvanic skin response (GSR), heart rate, respiration rate, and blood pressure has been shown to carry emotion-related information.It should be noted that the origin of such physiological signals primarily emanates from the peripheral nervous system; hence, these signals express emotion spontaneity, as they cannot be intentionally altered in contrast to facial expressions.
Lately, technological advances have allowed the sensing of some of these signals by smartwatches, employing, for example, photoplethysmography (PPG) to capture the respiration and heart rate, and detecting electrical pulses from the user's fingertip to the wrist to create a closed electrical circuit recording a single ECG strip (Apple watch).Figure 1 depicts examples of such data sensing approaches; more details can be found in [3].

EmotIon rEcognItIon
In order to infer an emotion from the analysis of the collected data, baseline emotion modeling, drawn from the field of psychology, needs to be first considered, such as discrete and dimensional models.In the discrete model proposed by Ekman [9], six universal emotions, i.e., fear, anger, joy, sadness, disgust, and surprise, transcending language, regional, cultural, and ethnic differences, are mainly considered.In dimensional models, multiple emotion dimensions are used for describing the emotions.In particular, Russel's most prominent 2D Circumplex model [10] includes the dimensions of valence (pleasantness, i.e., negative/positive) and arousal (activation level, i.e., low/high).
However, the fuzzy boundaries between discrete emotions and the complexity of the emotion dimensions impose obstacles for emotion labeling and recognition processes, requiring the adoption of advanced signal processing approaches to successfully overcome these.This need is more evident for the analysis of continuous emotion-related physiological signals (directly related with the SNGs), where their nonstationary nature, combined with measurement noise, hinder the underlying information driven by the emotions.In this vein, shifting from simple linear filtering to signal decomposition approaches (e.g., wavelet multiresolution analysis, empirical mode decomposition, swarm decomposition) separates the oscillatory modes of the signal at different frequency bands and facilitates communicating with the sensing systems and using intelligence, are needed to provide emotion outputs [5].wn triggers of feelings, expressions, and al responses, which their transitory synchronization the current emotional state, are games.In fact, game ics (e.g., story and characters, game mechanics and can be designed, in order to maximize emotion uring game play and allow different options for f the player's emotional and mental state [6].In this us NeuroGames (SNGs) have been recently where this nascent form of gaming involves brainnterfaces (BCI), such as electroencephalogram ology, instead of traditional controllers.Despite the ber of SNGs with limited number of game actions ailable in the market, SNGs have been developed ty of purposes, such as to improve learning and n skills, help people with Alzheimer's or Disease, Attention Deficit Hyperactivity Disorder Post-Traumatic Stress Disorder (PTSD), treat f chronic pain and depression, and facilitate stroke n [7].Examples of SNGs include the recent BrainAsssistant, Zukor Sports 1 Game Suite, along lder ones, e.g., Syncself 2, AmbuRun, NeuroRacer, .rticle, we aim to give the readers an overview of ensed emotions as adaptation controllers in humanmmunication, spanning from the state-of-the-art to proaches in this field.We propose a novel approach human-to-SNG communication is set and we way adaptation of the SNG characteristics can be e inferred emotions, towards achieving the desired ng SNG goals.The main contributions of this article placement of SNGs in the communication path of machine interaction, b) introduction of emotionesign framework of SNGs, c) integration of biosignals sensing and analysis that trigger apting feedback, and d) exploration of emotionation schemes of SNGs.To further exemplify the and design characteristics of such multisensed apted SNGs, we present a related application case, oSense that highlights the uptake propensity of wider community.ainder of this article, we first briefly describe the nsing and recognition landscape, examining the rce sensing and emotion inferring approaches.iscuss the SNGs design framework and the way NG communication is set.Next, we describe the tion of the SNG characteristics can be driven by the otions, towards achieving the desired characterizing .Then, we present EmoSense and evaluate its as a technological solution in the field of ck.Finally, we discuss the challenges and s in this area, followed by the main conclusions.
The human body is the locus of affective information; hence, emotion-related data can span from behavioral responses (e.g., facial expressions, social interaction) to physiological reactions (e.g., body signals alterations) [3].Considering the physical sensors embedded within mobile devices for sensing the external environment, their use as a data collection platform for unobtrusive sensing of emotion is natural.For example, smartphone keystroke dynamics provide digital biomarkers for detecting depressive tendencies [8].In addition, wearable devices, either commercial or customized (e.g., smartwatches, wristbands, smart rings, headset sensors) can acquire physiological signals, such as EEG, electrocardiogram (ECG), electromyogram (EMG), galvanic skin response (GSR), heart rate, respiration rate, and blood pressure has been shown to carry emotion-related information.It should be noted that the origin of such physiological signals primarily emanates from the peripheral nervous system; hence, these signals express emotion spontaneity, as they cannot be intentionally altered in contrast to facial expressions.
Lately, technological advances have allowed the sensing of some of these signals by smartwatches, employing, for example, photoplethysmography (PPG) to capture the respiration and heart rate, and detecting electrical pulses from the user's fingertip to the wrist to create a closed electrical circuit recording a single ECG strip (Apple watch).Figure 1 depicts examples of such data sensing approaches; more details can be found in [3].

A. Emotion Recognition
In order to infer an emotion from the analysis of the collected data, baseline emotion modeling, drawn from the field of psychology, needs to be first considered, such as discrete and dimensional models.In the discrete model proposed by Ekman [9], six universal emotions, i.e., fear, anger, joy, sadness, disgust, and surprise, transcending language, regional, cultural, and ethnic differences, are mainly considered.In dimensional models, multiple emotion dimensions are used for describing the emotions.In particular, Russel's most prominent 2D Circumplex model [10] includes the dimensions of valence (pleasantness, i.e., negative/positive) and arousal (activation level, i.e., low/high).
However, the fuzzy boundaries between discrete emotions the feature extraction that best distinguish between different emotional states by removing any noise effect.In fact, these extracted features capture the information embedded in the sensed signals related to the emotional states to identify, where non-relevant information, such as noise, is rejected.All these features consist the feature vector.Different machine and deep learning approaches have been proposed for emotion recognition by either using training and test sets from the extracted features (e.g., support vector machines, random forest, logistic regression, Bayesian networks) or by data-driven features learning (e.g., convolutional neural networks (CNNs), encoders, transformers, recurrent neural networks (RNNs)), respectively.To further facilitate emotion recognition, multisensed emotion-related data could offer better representation of the multi-dimensionality of emotional responses [3].Hence, a multimodal fusion can be adopted and applied to the feature-, model-, and decision-level [11].In other words, features from single-modality data are algorithmically used to form the combined feature vector for the classification (feature-level); or knowledge from learning a shared feature across multimodal data is combined to reveal interactions between the different data modalities (model-level); or decisions of multiple classifiers applied to multimodal data are combined into a common decision output (decision-level), mainly when it is difficult to combine the multimodal data in the same feature vector.

sErIous nEurogAmEs
SNGs are a part of the digital world and, apart from their entertaining role, they intend to achieve a so-called characterizing goal that is closely linked to the application area, yet, without compromising the player experience.For example, a breathing-based SNG can provide both entertainment and breathing control to achieve the characterizing goal of reduced anxiety.SNGs could be used in various ways, such as for health, brain training, learning, and change of behavior, attitude and enhancing knowledge.What is of major importance, though, is the balance between the serious and game aspects of the SNGs, along with efficient physiological recording, that could result in high-quality SNGs, as discussed next.

sngs dEsIgn FrAmEwork
Amongst the various SNG design frameworks, a game consumption framework that includes three basic components, i.e., mechanics, dynamics and aesthetics, the so-called MDA framework [12], is adopted here.The latter is followed since it involves aesthetics, i.e., the component directly related to emotions.More specifically, in the MDA, mechanics describes the game components at the level of representation and algorithms (e.g., how the games are constructed); dynamics describes the experience of the game, i.e., how the player interacts with the game mechanics during playing; and aesthetics describes the desired emotional responses of the participants when they interact with the game, and which is what makes the game fun.The MDA framework for SNGs development can steer the focus of the design for developing rules and mechanics that model cognitive and emotional processes.In this way, the game experience comes first, before the understanding of the underlying aims for gameplay, such as improving mental health or learning outcomes.
Within the MDA perspective, aspects that promote the quality in the serious and game parts of the SNGs, along with their balance, should be considered, resulting in an extended MDA (eMDA) framework; these include [13]: Quality Aspects in the Serious Part of the Game: • establishment of a positive experience and engagement during playing, by sustaining the game flow, ensuring varied gameplay, balance between skills and challenge, employing dynamic adaptation of game difficulty and complexity; fostering emotional connection and instinct arousal; • appropriate graphics and sound, ensuring clear interface and audio background/effects that promote the game tasks as part of the purpose of the game, area and target group.

3) Quality aspects in balance between serious and game parts:
• integration of the serious part within the gameplay by embedding the characterizing goal into the gameplay in a way that it cannot be avoided; adopting a co-creation approach including all related stakeholders (e.g., game designers, domain experts, users), via wireless (Fig. 2) or wired transmission) to the receiver side.Although connected systems are highly accurate, wireless systems are preferred, as wired BCI systems impose limitations on the users' movement and everyday living practices, whereas wireless transmission technology (e.g., Bluetooth, radio frequency) eliminates mostly such limitations.Wireless transmission has become an ubiquitous technology, allowing wireless BCI to be easily integrated into both acquisition, transmission and analysis of neurophysiological signals for portable and lightweight devices, that are easily and comfortably worn.The acquired data are then received by the signal processing and learning domain (Fig. 2).There, a data preprocessing step (e.g., data normalization, filtering, signal decomposition) is first applied, as the signals obtained by the BCI electrodes are generally contaminated by noise and/ or artifacts.These are usually a result of movements, poor electrode contact and quality, and displacement, power line interference, and even physiological interference from muscular, cardiac, ocular and respiratory activity.The preprocessed signals then follow a path of feature extraction, where they are represented by a feature vector that expresses measures of the underlying physiological activity of interest.Using machine learning, these features are translated into explainable classes and/or patterns.An alternative path is to apply explainable artificial intelligence (xAI) directly to the preprocessed data (Fig. 2) and arrive again at the explainable classes/patterns.The xAI system can be helpful in understanding of the reasoning behind a particular prediction or decision by a machine learning model.The estimated classes/patterns, feed, at the SNG domain, the normalization processes that maps these classes/patterns onto the eMAD framework, that translates the output into input controls of the SNG components.In this way, a sequence of changes within the SNG environment is evoked and materialized at the SNG interface as feedback (e.g., visual, auditory, text) to the user.Considering the latest developments in real-time embedded systems and advanced signal processing methods, practical implementation of (near) real-time sophisticated signal processing in, even, mobile devices is feasible, allowing for a smooth two-way humanto-SNG communication (Fig. 2) in fixed and/or mobile SNG interfaces.

muLtIsEnsEd EmotIon-bAsEd AdAptAtIon In sngs
Usually, the multisensed brain signals from the BCI system are processed (Fig. 2) for their time and/or frequency information from each sensor and/or the shared information between them.Here, we focus on the feature translation at the emotion domain, using the signals as control parameters for the adaptation process within the SNGs. Figure 3 illustrates the proposed adaptation scheme.In particular, the emotion recognition module (Fig. 2) provides the output of the dimensional and/or discrete emotion estimation.The latter is expressed in levels (low/high valence and arousal) and probabilities of the discrete emotion recognition, respectively.These are then normalized to the eMDA framework by correlating them with the SNG serious/game parts, in terms of their contribution to the characterizing goal, the progress, achievements (serious part), and positive experience, difficulty, emotional connection and engagement (game part).The assigned correlations are then used as control inputs to the SNG adaptation part; there, the dimensions of mechanics, dynamics, and aesthetics are altered accordingly, leading to changes at the SNG environment (external domain) level.The effect of the latter to the human-to-SNG communication is a measure of achievement of the SNG characterizing goal that is compared to the actually desired one.The minimization of the distance between the achieved and desired SNG characterizing goal feeds the control loop for optimizing the humanto-SNG communication.
The effect of the emotions on the eMDA dimensions can be implemented as a one-to-one correspondence (e.g., changing the SNG background color according to the valence levels; decreasing/ increasing the speed of the SNG objects based on low/high arousal levels); more complex relationships, however, can also be foreseen.
In fact, as the game mechanics is the driving force for the achievement of the desired emotional objectives or aesthetics of the SNG, through the invocation of the game, emotions could affect the transition from the core mechanics (i.e., the main SNG actions) to implied mechanics (i.e., SNG actions implied by the game genre) and additional mechanics (i.e., extra SNG actions that could differentiate games of the same genre).For example, shifting from fear to joy could unlock extra capabilities within the SNG, include more options and add more degrees of freedom to game objects.
Similarly, emotions could control the triggering of simple SNG dynamics (i.e., those that directly emerge from mechanics) towards complex dynamics (i.e., combined simple dynamics) and causal dynamics (i.e., dynamics with causal relationship).For instance, transition from negative to positive valence could evoke dynamics that allow opportunities to build or earn game items, change levels, modulate characters' behavior according to their previous status.
Emotions are intuitively connected with the SNG aesthetics and could be determinants of the shift from a simple aesthetic experience (e.g., sense-pleasure game) to more enhanced (e.g., make-believe game) and even immersive one (e.g., uncharted-territory game, self-discovery game).This will allow player's emotions to control the transition of the player to a different mental state of FIGURE 3. The proposed adaptation scheme of the SNG, driven by the features translation (Fig. 2) into the emotions' domain.
ed data are then received by the signal processing domain (Fig. 2).There, a data preprocessing step rmalization, filtering, signal decomposition) is first the signals obtained by the BCI electrodes are ntaminated by noise and/or artifacts.These are sult of movements, poor electrode contact and displacement, power line interference, and even l interference from muscular, cardiac, ocular and ctivity.The preprocessed signals then follow a path traction, where they are represented by a feature xpresses measures of the underlying physiological terest.Using machine learning, these features are nto explainable classes and/or patterns.An ath is to apply explainable artificial intelligence y to the preprocessed data (Fig. 2) and arrive again inable classes/patterns.The xAI system can be derstanding of the reasoning behind a particular r decision by a machine learning model.The lasses/patterns, feed, at the SNG domain, the n processes that maps these classes/patterns onto framework, that translates the output into input he SNG components.In this way, a sequence of thin the SNG environment is evoked and at the SNG interface as feedback (e.g., visual, t) to the user.Considering the latest developments mbedded systems and advanced signal processing ractical implementation of (near) real-time signal processing in, even, mobile devices is owing for a smooth two-way human-to-SNG ion (Fig. 2) in fixed and/or mobile SNG interfaces.SENSED EMOTION-BASED ADAPTATION IN SNGS e multisensed brain signals from the BCI system are g.2) for their time and/or frequency information from nd/or the shared information between them.Here, we feature translation at the emotion domain, using the trol parameters for the adaptation process within the e 3 illustrates the proposed adaptation scheme.In decreasing/increasing the speed of the SNG objects based on low/high arousal levels); more complex relationships, however, can also be foreseen.
In fact, as the game mechanics is the driving force for the achievement of the desired emotional objectives or aesthetics of the SNG, through the invocation of the game, emotions could affect the transition from the core mechanics (i.e., the main SNG actions) to implied mechanics (i.e., SNG actions implied by the game genre) and additional mechanics (i.e., extra SNG actions that could differentiate games of the same genre).For example, shifting from fear to joy could unlock extra capabilities within the SNG, include more options and add more degrees of freedom to game objects.
Similarly, emotions could control the triggering of simple SNG dynamics (i.e., those that directly emerge from mechanics) towards complex dynamics (i.e., combined simple dynamics) and causal dynamics (i.e., dynamics with causal relationship).For instance, transition from negative to positive valence could evoke dynamics that allow opportunities to build or earn game items, change levels, modulate characters' behavior according to their previous status.
Emotions are intuitively connected with the SNG aesthetics and could be determinants of the shift from a simple aesthetic experience (e.g., sense-pleasure game) to more enhanced (e.g., make-believe game) and even immersive one (e.g., unchartedterritory game, self-discovery game).This will allow player's Fig. 3.The proposed adaptation scheme of the SNG, driven by the features translation (Fig. 2) into the emotions' domain.
flow, where, during playing, s/he is immersed in the SNG with focus, involvement and enjoyment.This is further supported by the way emotions underpin function of the human brain, as affective behavior is the outcome of a multilevel integration of both physical and psychological processes [6].Keeping the player in a state of flow, scaffolds the convergence towards the achievement of the desired characterizing SNG goal.
In the implementation of the emotion-based SNG adaptation, the differences in the affect dynamics across the users should be considered.The latter refer to their emotional granularity (i.e., ability to differentiate between various positive or negative discrete emotions), emotional inertia (i.e., how much positive or negative affect carries over across successive moments), and emotional instability (i.e., the average change in emotional intensity between two successive measurement occasions for positive or negative affect) [2].This fosters the role of personalized design in the emotion-based SNG adaptation.For example, players with low emotional granularity cannot provide high variety within the discrete emotions; hence, in such a case, the valence/arousal domain should be preferred for feature translation guiding the SNG adaptation (Fig. 3).In addition, the time window (e.g., 5-10s) needed for the estimation of the emotions from the acquired multisensed data should be harmonized with the time duration needed for the SNG adaptation to take place in a smooth way, complying with the eMDA framework.

AppLIcAtIon cAsE And AccEptAncE EvALuAtIon
When designing a human-to-SNG communication setting under a co-creation approach, its acceptance from stakeholders (e.g., SNG game developers, researchers, field experts, and users), should be considered.Here we present an emotion-adapting SNG system, namely EmoSense, and evaluate its acceptance from different involved stakeholder groups.This process entails the user affordance and subsequent interaction within the SNG that, via cognitive and emotional processes, could lead to affordance actualization.In this way, SNG is shaped by the users' affordance and perceptions to a large extent, and at the same time, shape the SNG environment in a dynamic way [14].

EmosEnsE ArcHItEcturE
The main parts of the EmoSense architecture are illustrated in Fig. 4. In particular, a customized five-modality/eight-channel headset is used as the signal acquisition and wireless transmission system, manufactured by PLUX Biosignals (Fig. 4a).It consists of three EEG channels, two functional Near InfraRed Spectroscopy (fNIRS) channels, one Electro-Dermal Activity (EDA) channel, one Blood Volume Pulse (BVP) channel, and one temperature (T) channel.All data are acquired with a sampling frequency of 500Hz and sent via Bluetooth to the hosting unit, which, in the current design, is a 5G smartphone (minimum computational requirements: Memory 256GB 8GB RAM, CPU Octa-core, GPU 8-12 GB, Bluetooth  4.1).In the latter, signal preprocessing and decomposition (i.e., multivariate swarm decomposition), combined with pre-trained deep learning (i.e., CNN) are used to infer the emotion estimation, considering the contribution of the different sources via a model-and decision-level fusion (Fig. 4b).The estimated emotions drive two SNG games (Flame, GlassGlobe), designed in Unity 3D within the neurofeedback and eMDA framework, sharing the same desired characterizing goals, i.e., increase of player's focus (centering of attention on a stimulus), concentration (focus on one thing for a continuous period), and relaxation (low tension emotional state with an absence of arousal) (Fig. 4c).
The mechanics in the Flame SNG, i.e., igniting and sustaining the flame of an oil lamp, its dynamics (e.g., sustain the flame despite different wind integration of both physical and psychological processes [6].
Keeping the player in a state of flow, scaffolds the convergence towards the achievement of the desired characterizing SNG goal.
In the implementation of the emotion-based SNG adaptation, the differences in the affect dynamics across the users should be considered.The latter refer to their emotional granularity (i.e., ability to differentiate between various positive or negative discrete emotions), emotional inertia (i.e., how much positive or negative affect carries over across successive moments), and emotional instability (i.e., the average change in emotional intensity between two successive measurement occasions for positive or negative affect) [2].This fosters the role of personalized design in the emotion-based SNG adaptation.For example, players with low emotional granularity cannot provide high variety within the discrete emotions; hence, in such a case, the valence/arousal domain should be preferred for feature translation guiding the SNG adaptation (Fig. 3).In addition, the time window (e.g., 5-10s) needed for the estimation of the emotions from the acquired multisensed data should be harmonized with the time duration needed for the SNG adaptation to take place in a smooth way, SNG game developers, researchers, field experts, and users), should be considered.Here we present an emotion-adapting SNG system, namely EmoSense, and evaluate its acceptance from different involved stakeholder groups.This process entails the user affordance and subsequent interaction within the SNG that, via cognitive and emotional processes, could lead to affordance actualization.In this way, SNG is shaped by the users' affordance and perceptions to a large extent, and at the same time, shape the SNG environment in a dynamic way [14].

A. EmoSense Architecture
The main parts of the EmoSense architecture are illustrated in Fig. 4. In particular, a customized five-modality/eight-channel headset is used as the signal acquisition and wireless transmission system, manufactured by PLUX Biosignals (Fig. 4(a)).It consists of three EEG channels, two functional Near InfraRed Spectroscopy (fNIRS) channels, one Electro-Dermal Activity (EDA) channel, one Blood Volume Pulse (BVP) channel, and one temperature (T) channel.All data are acquired with a sampling frequency of 500Hz and sent via Bluetooth to the hosting unit, which, in the current design, is a 5G smartphone (minimum computational requirements: Memory When designing a humanto-SNG communication setting under a co-creation approach, its acceptance from stakeholders (e.g., SNG game developers, researchers, field experts, and users), should be considered.
intensity levels) and aesthetics (e.g., alteration in background scene/color of the objects, added background audio), are all controlled by the player's emotions (Fig. 4c).Similarly, in the GlassGlobe SNG, emotions control the construction of a temple within a glass globe (mechanics) with gradually increasing the scale, number and type of construction items (dynamics) and creating various versions of the background setting, temple color and addition of sound effects (aesthetics) (Fig. 4c).

EmosEnsE tEcHnoLogy AccEptAncE EmpIrIcAL study
As with any technology, especially in the BCI setting, uptake and continued use by the user is most important.In order to evaluate the technology acceptance of the EmoSense by the relevant stakeholders, an empirical study was conducted.
In the latter, an online survey was designed and developed by a multidisciplinary team, engaging mental health experts, doctors, health professionals, researchers, game designers/developers, and end-users.The empirical study aimed at capturing the beliefs of the stakeholders upon the proposed EmoSense as a novel conceptual solution (no actual use of it was involved).
Survey structure: The survey included participant demographic information and feedback on the acceptance of EmoSense, by providing detailed information for its structure, functionality and potential use.The well-established Technology Acceptance Model (TAM) [15] was adopted, and the survey feedback was constructed as fivepoint Likert scale questions based upon the TAM constructs of Perceived Ease of Use (PEoU), Perceived Usefulness (PU), Usage Attitude (UA), and Intention to Use (IU).
Data collection and sample characteristics: The survey took place between 1/8-15/11/2022, using email-based recruitment; 300 invitations targeting the stakeholder groups were sent distributed, describing the survey and inviting online participation.Non-probability sampling design was followed, using convenience sampling, i.e., each respondent was selected for inclusion in the sample based on the ease of access.A total of N = 106 participants provided online consent and fully completed the online survey.Their demographics include: EmoSense technology acceptance evaluation results: Figure 5 depicts the estimated path coefficients between the TAM constructs, along with their significance (p values).From Fig. 5 it is evident that the IU mainly depends on the UA, which, in turn, is affected by the PU and PEoU.Moreover, PU is affected by the PEoU; yet, it seems to have less effect on IU.Additionally, Table 1 tabulates the consistency and validity analysis results, from where it is clear that all selected constructs are reliable (all Cronbach's a and CR values >0.70; all AVE values >0.50).Grouping analysis based on gender, age, education and occupation, did not reveal any significant effect on the path coefficients.These results denote that the EmoSense was well perceived by the stakeholders in terms of its PEoU, PU, UA towards the IU, validating the TAM framework.

cHALLEngEs And opportunItIEs
Emotions are dynamic and complex in nature, usually not experienced in the pure form of a single emotion; hence, the coexistence of multiple basic emotions at one time poses difficulties in emotion recognition.The acquisition and advanced analysis of multimodal signals, combined with knowledge from neuroscience, could provide more insights about the emotion expression in the brain, assisting further with their accurate recognition.Subjectivity in experiencing and expressing emotions reveal the need for personalization and adaptation in the emotion inference system.Augmenting the acquired biosignals with the personal affect dynamics, the emotion inference models could enhance their performance, providing interpretation of the emotional changes for a specific user.Moreover, the disruption in humanto-AI communication from AI-based chatbots (e.g., ChatGPT, GPT-4, and Bard), has opened up additional dimensions of involving such AI FIGURE 5.The adopted TAM framework [14] for the acceptance of the Emo-Sense, with the interconnected constructs and their standardized path coefficients and p values (in parentheses), estimated using PLS-SEM Bootstrap analysis of 200 realizations [15]; ***: p < 0.001.
with gradually increasing the scale, number and type of construction items (dynamics) and creating various versions of the background setting, temple color and addition of sound effects (aesthetics) (Fig. 4(c)).

B. EmoSense Technology Acceptance Empirical Study
As with any technology, especially in the BCI setting, uptake and continued use by the user is most important.In order to evaluate the technology acceptance of the EmoSense by the relevant stakeholders, an empirical study was conducted.In the latter, an online survey was designed and developed by a multidisciplinary team, engaging mental health experts, doctors, health professionals, researchers, game designers/developers, and end-users.The empirical study aimed at capturing the beliefs of the stakeholders upon the proposed EmoSense as a novel conceptual solution (no actual use of it was involved).
Survey structure: The survey included participant demographic information and feedback on the acceptance of EmoSense, by providing detailed information for its structure, functionality and potential use.The well-established Technology Acceptance Model (TAM) [15] was adopted, and  TABLE 1. PLS-SEM -analysis results for the TAM constructs (Fig. 5) of the EmoSense acceptance study.engines in the design framework of the SNGs, under a collaborative argumentation between the designer and AI engines.Thus, internal SNG design thoughts could be shared with such AI chatbots, especially within the context of anthropomorphism and explainability.Considering the trustworthy hybrid decision-support, mixed and sliding decision-making could take place, assisting context interpretation, dealing with uncertainty, transparent anticipation, reliability, interdependencies, and promoting augmented decision-making during the SNG design.
In the human-to-SNG communication, personal data of the user are involved, as the human brain's neuronal activity is accessed, recorded, interfered or modified.This poses data privacy challenges and invokes considerations for the protection of cognitive freedom, mental privacy, mental integrity and psychological continuity.This can be dealt with by adopting data protection regulations (e.g., EU GDPR), following privacyand security-by-design development, combined with ethics, transparency and explainability in the emotion inference mechanisms (e.g., xAI), and knowledge transfer from large-scale accredited biobanks, establish a trustworthy framework for SNG development, acceptance and use.
Recently, there has been a notable shift towards immersive experiences in human-to-machine communication, with huge investments in the emerging markets (e.g., Metaverse, NeuroGames).Following this trend, new growth possibilities of the SNGs market are anticipated, due to rising user awareness and increased adoption of advanced gaming (neuro)technologies providing real-time brain monitoring [14].This is further enhanced by the beneficial potential of SNGs in many areas, such as health and learning, merging the emotion domain activity with games.Integration of digital in everyday life, along with evolving digital consumption applications in a wider context where gaming could be seen as the new digital paradigm, sustains the current evolutionary stage of SNGs towards new forms of multi-playing and communicating under shared emotional states.

concLusIon
In this article, we provide an overview of the way emotions can be used within the human-to-SNGs communication, exploring the aspects of emotion sensing and inference, emotion-adapting SNGs design framework and adaptation mechanisms.We present an application case of SNG, namely EmoSense, in the field of neurofeedback to demonstrate its practicality and usefulness in skills improvement, presenting the experimental results from an acceptance evaluation study.We also discuss the challenges and opportunities of SNGs, fostering the parallel advancements in emotions sensing, privacy regulations, personalization and new emerging application scenarios of SNGs.
Sofia B. Dias, Herbert F. Jelinek, and Leontios J. Hadjileontiadis Sofia B. Dias (corresponding author) is with Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates and also with the Centro Interdisciplinar de Performance Humana (CIPER), Faculdade de Motricidade Humana-Universidade de Lisboa, Portugal; Herbert F. Jelinek is with Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates; Leontios J. Hadjileontiadis is with Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates and also with Aristotle University of Thessaloniki, Greece.

FIGURE 1 .
FIGURE 1. Examples of emotion-related data sources acquired from the human body using different sensors.

Fig. 1 .
Fig. 1.Examples of emotion-related data sources acquired from the human body using different sensors.

FIGURE 4 .
FIGURE 4. The architecture of the EmoSense application case: a) the multimodal signal acquisition system; b) the biosignals processing and emotion recognition module, incorporating swarm decomposition and CNN models fusion resulting in the emotion estimation; and c) the interface of the two SNGs (Flame and GlassGlobe), with the screenshots showing their emotion-based structural adaptation.

Fig. 4 .
Fig. 4. The architecture of the EmoSense application case.(a) The multimodal signal acquisition system, (b) the biosignals processing and emotion recognition module, incorporating swarm decomposition and CNN models fusion resulting in the emotion estimation, and (c) the interface of the two SNGs (Flame and GlassGlobe), with the screenshots showing their emotion-based structural adaptation.
Data analysis: The Partial L Modeling (PLS-SEM) with evaluate the standardized path constructs, their significance (Cronbach's >0.7),and c reliability (CR)>0.70;average using SmartPLS 4.0 and Stata EmoSense technology ac Figure 5 depicts the estimate TAM constructs, along with th Fig. 5 it is evident that the IU m in turn, is affected by the P affected by the PEoU; yet, it Additionally, Table I tabulat analysis results, from wher constructs are reliable (all Cro all AVE values >0.50).Groupi education and occupation, did on the path coefficients.These was well perceived by the sta PU, UA towards the IU, valid VI.CHALLENGES Emotions are dynamic and experienced in the pure form coexistence of multiple basi difficulties in emotion recognit