IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 3, NO. 3, JULY-SEPTEMBER 2012 285 Co-Adaptive and Affective Human-Machine Interface for Improving Training Performances of Virtual Myoelectric Forearm Prosthesis Iman Mohammad Rezazadeh, Mohammad Firoozabadi, Huosheng Hu, Senior Member, IEEE, and S. Mohammad Reza Hashemi Golpayegani Abstract—The real-time adaptation between human and assistive devices can improve the quality of life for amputees, which, however, may be difficult to achieve since physical and mental states vary over time. This paper presents a co-adaptive humanmachine interface (HMI) that is developed to control virtual forearm prosthesis over a long period of operation. Direct physical performance measures for the requested tasks are calculated. Bioelectric signals are recorded using one pair of electrodes placed on the frontal face region of a user to extract the mental (affective) measures (the entropy of the alpha band of the forehead electroencephalography signals) while performing the tasks. By developing an effective algorithm, the proposed HMI can adapt itself to the mental states of a user, thus improving its usability. The quantitative results from 16 users (including an amputee) show that the proposed HMI achieved better physical performance measures in comparison with the traditional (nonadaptive) interface (p-value < 0:001). Furthermore, there is a high correlation (correlation coefficient < 0:9; p-value < :01) between the physical performance measures and self-report feedbacks based on the NASA TLX questionnaire. As a result, the proposed adaptive HMI outperformed a traditional HMI. Index Terms—Human-machine interface, affective measure, forehead bioelectric signals, prosthetics, real-time adaptation, virtual reality Ç 1 INTRODUCTION 1.1 Background rapid engineering and biomedical advances are making a range of myoelectric devices available to restore or augment the hand functions of amputees [1], [2], [3], [4], [5], [37]. These devices aim to restore lost function and leverage the remaining intact aspects of the physiology and anatomy of user hands. However, it is clear from many studies that about 30 to 50 percent of upper extremity prosthesis users prefer to use a simple mechanical prosthesis instead of a myoelectric one as these myoelectric devices are difficult to use at the moment [6], [7], [8]. Assuredly, a myoelectric prosthesis is a potentially useful device that can enhance the quality of life for a user T ODAY, . I.M. Rezazadeh is with the School of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran. E-mail: [email protected]. . M. Firoozabadi is with the School of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran and with the School of Medical Sciences, Tarbiat Modares University, Tehran, Iran. E-mail: [email protected]. . H. Hu is with the School of Computer Science & Electronic Engineering, University of Essex, Colchester CO4 3SQ, United Kingdom. E-mail: [email protected]. . S.M.R.H. Golpayegani is with the School of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran and with School of Biomedical Engineering, Amir Kabir University of Technology, Tehran, Iran. Manuscript received 18 Mar. 2011; revised 21 Nov. 2011; accepted 18 Jan. 2012; published online 6 Feb. 2012. Recommended for acceptance by S.-W. Lee. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number TAFFC-2011-03-0025. Digital Object Identifier no. 10.1109/T-AFFC.2012.3. 1949-3045/12/$31.00 ß 2012 IEEE by assisting him/her in performing activities that are otherwise impossible [9], [10]. However, most of the concerns are coming from the prosthesis’s usability issues such as 1) physical interaction: including mechanical, electrical, and sensory elements; 2) cognitive interaction: including control/manipulation schemes, and 3) interactive interfacing [11], as well as its ability to function in a natural and intuitive manner. This paper is focused on the control scheme and interface of a myoelectric prosthesis. The control scheme provides proper outputs for mechanical parts according to its inputs. At the same time, the prosthesis reacts to the output commands, thereby affecting the perceptual cognition process of users [11]. There are four important considerations when designing the prosthesis control scheme to retain the robustness and stability of its control unit over a long period of operation: the number of distinct movement classes to classify and the accuracy of the classifier, 2. the intuitiveness of the actuating controller, 3. the response time, and 4. the online training of a classifier and its adaptability to cope with pattern variations [10], [11], [12], [13], [14]. As we know, without the online/real-time adaptation a control system could be faced with exponentially rising error over a long-run operation, and its performance will degrade. Momen et al. [13] stated that it is not clear whether the previously reported myoelectric classifiers have realtime adaptation ability to deal with variations in myoelectric signals which come from physiological factors (for example, sweating and fatigue, which exhibit gradual 1. Published by the IEEE Computer Society 286 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, changes) or physical factors (such as electrode displacement). Kato et al. [12] controlled an EMG prosthetic hand by employing an adaptable neural network which can manage data learning by examining the mapping to a training set of data in real time. They claimed that their proposed system could cope with gradual and drastic changes in the mapping. Fukuda et al. [16] used the EMG entropy level as a measure of the classifier input-output pairs’ validity. They stated that if the developed EMG entropy was lower than a predefined threshold, then the reliability of the classified patterns could be high. Thus, the input-output pairs could be added to the online training set of neural networks while the oldest pairs were deleted from it. 1.2 Virtual Reality as Training Medium for Interactive Interfacing Research works in [48], [49], [50], [51], and [52] showed that if a new interface differs significantly from the initial training condition, then a new scheme in a mental model is formed to lead to different synchronization patterns in the brain, i.e., more errors and confusion. According to recent studies [17], [37], Virtual Reality Environment (VRE) technology provides adaptable and rich media to create conditions for the assessment and training of motor deficits. Enhanced VRE technology is currently being explored in prosthetic limb training. It has been shown that learning in VRE can form a robust mental model to perform other ADLs [38]. In addition, VRE can enable its users to make mental leaps across distinct conceptual domains which are especially important in the initial training phase of using a myoelectric prosthesis. 1.3 Mental Workload, Level of Difficulty, and Emotion Mental workload can be defined as an intervening variable similar to attention that modulates or indexes the tuning between the demands of the environment and the capacity of the operator (user) [26], [39]. This highlights the two main parameters of workload: the capacity of operators and the task demands made on them. The workload increases when the user’s capacity decreases or the task demands increase. Both the capacity and task demands are unfixed entities and are affected by many factors. According to [39], an interface’s operator can experience different levels of workload depending on the task-demands. However, the user’s performance does not necessarily decline when the operator experiences a high workload. One can keep performing on a maximum level by increasing effort. Nonetheless, problems can arise when this effort is required for a long period. The physiological data to understand the effort from the operator are used in various studies (e.g., [31], [32], [33]). Specifically, electroencephalography signals have yielded a reliable description of the cognitive state [17], [34]. The Alpha range (8-13 Hz) of a frontal EEG (electroencephalography) signal can function as a moderator/mediator of the user’s cognitive states and determine the emotional responses and mental workload while performing a task [40]. It represents the oscillations of postsynaptic potentials in the nerocortex activity and is modulated by semantic memory processes. It has also been shown that this subband has a relationship with attention demands and mental states VOL. 3, NO. 3, JULY-SEPTEMBER 2012 (for example, level of alertness, expectancy, mental relaxation, and satisfaction) during task performance [47]. Furthermore, Liu et al. suggested that the Alpha level correlated significantly with the cognitive abilities to cope with tasks [47]. The activity within the Alpha range may be inversely related to underlying cortical processing since decreases in Alpha tend to be observed when the underlying cortical systems engage in moderate to difficult tasks [40]. Therefore, there is a negative correlation between brain activity (mainly Alpha range) and performance indices under a workload [40]. Matsunaga and Nakazawa [41] developed an adaptive HMI to enhance the user’s satisfaction using an EEG signal in a closed loop. They measured the satisfaction level by employing Fast Fourier Transformations (FFT) to determine the power spectrum of an EEG. They concluded that there are two bands: alpha, which is proportional to satisfaction, and beta, which is inversely proportional to satisfaction. In addition, the right front is related to positive emotions, while the left front is related to negative emotions. The amplitude of alpha increases when the user becomes more certain about the task and feels more satisfaction about it. According to some studies, the statistical entropy could be employed as a measure of the system complexity and the degree of order/disorder of a signal, and it provides useful information about the underlying dynamical process associated with the signal [30], [31], and [35]. The EEG entropies correlate closely with cortical activity. For example, in performing a task the EEG entropy increases in proportion to the mental workload and level of difficulty [43], [44]. Thus, one could assume that when a subject is burdened by a normal mental workload, the degree of disorder of the alpha subband is less than for a more complex/difficult task that requires a greater mental workload. In this study we focused on the initial training stages by using a virtual myoelectric prosthesis. At these stages, increasing the entropy of alpha wave reflects the level of difficulty and mental workload within that stage. Assessing and controlling this measure is an important process to maintain the level of involvement and pleasure for the user [56], [57]. Analyzing EEG signals revealed that there exist direct associations between workload and level of pleasure for a specific task [53]. It should be noted that task performing at different levels of difficulty induces one of three emotional states: boredom, engagement, and anxiety [54], [55], [56], [57]. For example, since the skill increases, the subject may switch from the engagement state to the boredom state, or, in case of initial training for a new task, if the task requires great mental workload, then it may be difficult for the subject to cope with the task during a long run, and consequentially reduce the level of engagement. As a result, the entropy of the alpha subband can be considered as an affective measure (cue) that mirrors the mental workload, levels of difficulty, and , consequenly, the user’s emotional state and can be used to enhance the interaction level within an HMI. 1.4 Study Goal An individual’s emotion can affect his or her performance within a human-machine interface. Therefore, it would be a key feature for an HMI to estimate or predict the individual’s REZAZADEH ET AL.: CO-ADAPTIVE AND AFFECTIVE HUMAN-MACHINE INTERFACE FOR IMPROVING TRAINING PERFORMANCES OF... 287 Fig. 1. General block diagram of the proposed method and control unit as its core. emotional state for improving service quality [45]. To solve the mentioned shortcomings in the cognitive interaction of prosthesis’ interface, we hypothesized that using the subject’s emotional indices for updating the prosthesis controller’s scheme within an interactive medium could enhance the interface and consequently improve the user’s performance. Thus, we designed and implemented a coadaptive and affective human-machine interface (caHMI) and updated the interface control scheme using the user’s mental states. The manipulating commands for controlling a virtual forearm were extracted from the biceps and triceps activities of a subject. Using a pair of electrodes placed on the frontal facial region of a subject, the relative bioelectric signals were recorded to explore the subject’s affective measures. Here, we would like to clarify the relationship between the mental workload, task demands, and performance. We hypothesize that by employing the subject’s affective cues, the system’s context awareness and interactivity will increase. Thus, the usability of the HMI will be enhanced compared to conventional HMIs, which will increase the usability and performance of the interface. 2 SYSTEM DESIGN AND ARCHITECTURE Fig. 1 shows the block diagram of the proposed co-adaptive and affective HMI (caHMI) that consists of three units: Virtual Forearm Manipulation (VFM), Affective Cues Extraction (ACE), and the control unit. The virtual forearm is controlled by user’s manipulating commands recorded from biceps and triceps muscles using the VFM scheme. Meanwhile, based on the ACE scheme, the control unit modifies and updates the VFM’s inference parameters using the user’s affective measures extracted from the bioelectric signals. 2.1 Site Selection and Placement of Electrodes As illustrated in Fig. 1, three pairs of pregelled Ag/AgCl electrodes were placed on the subject’s upper arm muscles and the frontal region of the subject’s head in a differential configuration to obtain the highest signal amplitude. Channel 3: One pair of electrodes was placed on the subject’s forehead region, close to the Fp1 and Fp2 regions in the international 10-20 system [22]. As stated in Section 1.3, this channel is responsible for extracting the affective measures and cues based on the ACE scheme [18], [19], [20], [21], [22] (see Section 2.6). It should be noted that the interline of the electrodes of this channel is somehow perpendicular to the frontalis muscle fibers. Thus, the effects of the myoelectric signals that are generated during frowning or pulling up the eyebrows are minimized on this channel. In addition, the most dominant frequency band of the EMG is above 30 Hz, which is about three times greater than the dominant frequency range of the alpha subband. It was found that the covarying effects of the EMG frequencies at the sites where the alpha frequencies were recorded did not alter the pattern of results, nor did the covarying asymmetries in alpha band activity resulting from contractions of the frontalis and temporalis muscles. These findings suggest that the asymmetries in the EMG band were unlikely to have caused the observed Alpha asymmetries. In addition, a facial EMG cannot be artifactually localized as brain activation in functional imaging [46]. Thus, this channel configuration was a good choice for recording 288 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, alpha subband frequencies with low contamination from facial electroencephalography signals. 2.2 Data Acquisition Setup and Participants In this research, a Biopac system (MP100 model and ack100w software version) [23] was used to acquire bioelectric-signals and was connected to a PC (1.73 GHz, 2 GB RAM) for further processing. The sampling frequency is selected at 1,000 Hz and the amplifier gain is 5,000, respectively. Sixteen volunteers participated in this study to validate the experimental procedure and the robustness of the proposed method. All of the subjects gave informed consent to participate and the ethical guidelines were followed in the conduct of the study. . . A1-15: Fifteen male nonathletic and healthy adults, aged 19-30 with no upper limb deficiencies. All of the volunteers were from the School of Biomedical Engineering, Azad University in Tehran, Iran. D1: A 26-year old adult with the left forearm amputated without previous experience on myoelectric control was recruited from the public. It was particularly desirable to choose an amputee who was not a powered prosthesis user because he would most likely be unfamiliar with arm muscle control. It is generally recognized that reliable EMG control necessitates user training. Therefore, the lack of prior training in our selected participant might present a nontrivial challenge to our control method. 2.3 Offline Data Recording Protocol In each recording session, the volunteer was asked to sit on a comfortable chair. For a healthy subject, if he was righthanded, his left hand was selected to be used in the experiment and vice versa. Then, the prompt forearm and wrist were fixed using an adhesive strap to prevent movements in the elbow and wrist joints. Thus, the collected myoelectric signals from the subject’s biceps and triceps muscles could be considered as the consequences of isometric contractions. In addition, the amplitude of the EMG signal was proportional to the force produced by the muscle. It should be noted that for subject D1, due to the left forearm amputation, his left arm was selected for recording the signals. Before each recording session, the volunteer was trained to generate two different isometric myoelectric signals using his biceps and triceps muscles. Then, he was asked to take a rest and try to relax for a period of 5 minutes. After this period, the quiescent bioelectric signals from all three data channels were recorded for a period of 1 minute, while he was still resting. These quiescent signals were used to determine the on-set threshold to distinguish between the rest (no-action) and active states of the EMG classifier and also determine the baseline for estimating the mental workload. Then, the volunteer was asked to perform one of the mentioned isometric contractions moderately with respect to maximum voluntary contraction level for each trial. The recording period in each trial was started 1 s after the beginning of the movement—to eliminate the transient effect of the EMG—and ended right after 2 s from the beginning of the recoding. After a 10 s rest, he was asked to repeat the movement again. The above movement-rest task VOL. 3, NO. 3, JULY-SEPTEMBER 2012 was cycled 10 times. The resting period was chosen empirically to eliminate the fatigue effect during training. For each subject, the recording session took about 10 minutes based on the above protocol. 2.4 Data Preprocessing A value of 0.1 Hz was chosen as the low cutoff frequency of the Butterworth filter to avoid motion artifacts, and a narrow band-stop Butterworth filter (48-52 Hz) was also used to eliminate the line noise and were applied to the raw data. Then, the recorded data from Channels 1 and 2 were passed through parallel Butterworth digital filter banks with frequency characteristics from 30 to 450 Hz to obtain the main power spectrum of EMG signals [10], [58]. These signals were used to establish the physical interaction between the virtual forearm and the user. Furthermore, a band-pass filter (8-13 Hz) was employed on the raw data from Channel 3 to select the EEG Alpha range. 2.5 Data Processing in Virtual Forearm Manipulation (VFM) Unit Recent studies [10], [13], [14], [15] have explored continuous prosthesis control strategies using a variety of algorithms for feature set extraction, dimensionality reduction, and classifier architecting. In this section, the data processing components in the VFM unit are described. 2.5.1 Data Segmentation Because of the real-time approach, an adjacent segment length plus the processing time for generating the classifier’s commands should be equal to or less than 300 ms [10]. Thus, by considering all of the above factors and referring to some detailed studies, a nonoverlapped segment length of 256 ms for Channels 1 and 2 was chosen in experiments [10]. 2.5.2 Feature Selection and Onset Detection In this study, by considering the simplicity of the RMS extraction process and its accuracy, the RMS of EMG signals ðRi Þ from Channels 1 and 2 (EXGi ) were calculated within a nonoverlapping window of 256 ms [10]: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi RT 2 0 EXGi dt ; ð1Þ Ri ¼ RMSðEXGi Þ ¼ T i ¼ 1 or 2: Channel Number T ¼ 256: The onset value (RMST hri ) for each movement was determined as the point when the RMS features were greater than the average of the RMS of the quiescent signal plus three times its standard deviation (std) [14]: RMS T hri ¼ fRi  3 MeanðRMSðEXGQuiescent ÞÞ þ 3 stdðRMSðEXGQuiescent ÞÞg: ð2Þ Then, the RMS features, which are greater than the threshold, were normalized (Si ) using the same method as in [16] and the RMS features less than threshold are considered as “No Action”: Si ¼ RMS T hri  MeanðRMSðEXGQuiescent ÞÞ : K i¼1 ðRMS T hri  RMSðEXGQuiescent ÞÞ ð3Þ REZAZADEH ET AL.: CO-ADAPTIVE AND AFFECTIVE HUMAN-MACHINE INTERFACE FOR IMPROVING TRAINING PERFORMANCES OF... Furthermore, to have a more separable feature space, all the extracted features can be transformed to a nonlinear simple feature space using a logarithm transform function (log) to spread the concentrated data points while condensing the highly scattered points [13]: Fi ¼ log ðSi Þ: ð4Þ 2.5.3 Classification In accord with the studies [18], [19], [24], [25], the inputoutput subtractive fuzzy clustering method (SFCM) was chosen as the classification approach here to obtain a set of initial rules for the fuzzy inference system. Then, an Adaptive Neuro-Fuzzy Inference System (ANFIS) was employed for adjusting the obtained inference system’s parameters. ANFIS has shown its capabilities as a powerful classifier in the movement classification area [27]. Here in this study, after performing the initial recording session for each subject, the extracted features from trials with the odd index numbers and even index numbers were added to the SFCM inference system’s training and the testing sets, respectively. Then, ANFIS was employed for adjusting the obtained SFCM inference system’s parameters because of the high training speed and robustness of this combined method. The obtained inference system is capable of discriminating forearm flexion and extension movements. The above training procedure occurred only one time for the offline recording data to obtain initial controller; however, the same was applied in the online and continuous classifier updating. It should be noted that, in the online operation, the majority voting (MV) is applied as a postprocessing method to manage excessive classified output regarding continuous segmentation. Here, MV includes the last and next mð¼ 10Þ-decisions for a given point to generate a new decision. 2.6 Data Processing in Affective Cue Extraction Unit By considering the factors mentioned in Sections 1.3 and 2.4, the acquired data from channel 3 was passed through an 8-13 Hz band-pass filter to select the EEG Alpha range. The filtered data from Channel 3 was divided into nonoverlapped time slots of 128 ms. Then, the logarithm of energy entropy (HLogEn ) (hereafter, entropy or statistical entropy) for each time slot is calculated by HLogEn ¼  N 1 X ðlog2 ðPi ðxÞÞÞ2 ; ð5Þ i¼0 where x is the alpha band data samples, N ¼ 128, and P ðxÞ is the probability distribution function of x [31]. 2.7 Online Adaptation in Control Unit There are two critical considerations in the classifier online adaptation: 1) the recognition and updating of valid training data, and 2) applying the adaptation algorithm during operation. Training data require clarification as to whether the classified patterns coincide with the user’s intention. Therefore, the input-output pairs of classifier must be monitored, and their reliability should be examined continuously to update the training data. In addition, applying 289 an online adaptation algorithm to a classifier during operation requires a distinguished method [10]. In this study, the control unit is the core of the proposed HMI, and it modifies the inference system of VFM according to the ACE’s outputs. The initial setup of the control unit is achieved by deploying input-output data recorded in the offline data recording protocol—and the same methods described in Sections 2.4 and 2.5. In the online adaptation phase, the control unit monitors the average HLogEn of the Alpha range of a user within a predefined period (TTM: Time to Monitor) during the experiment. Based on the discussion in Sections 1.3 and 2.6, if the average entropy measure is below the predefined threshold within the TTM period, then the dimensional complexity and degree of disorder in the alpha subband is low. Thus, it can be concluded that the user has an affordable mental workload while performing the requested task. Because the reduction of the mental workload of a user and an increase in performance are implied by the HMI, the classifier’s performance (outputs) can be considered to be reliable and valid. Then, Algorithm I (see Fig. 2) is applied to the valid input-output pairs. It is also very important to obtain the dynamics and backgrounds of the input patterns of VFM and employ them in the updating procedure of the control unit. Thus, Algorithm I decides whether to save the input’s history by calculating the euclidean distance between the old and new input data. If the old inputs are within the predefined euclidean distance from the new inputs and the outputs are the same, then the old data will be appended into new inputoutput pairs. In general, the value for the valid distance is determined by the accuracy needed for the task. The valid distance value is chosen between 0.01 to 0.05 in the normalized feature space. However, it should be mentioned that appending old data to new ones may cause data redundancy and requires more storage space. Because the training time for the inference system increases in proportion to the size of the valid data set IP, the size of the data should not be too large to cope with real-time constraints. To resolve the data size problem an importance factor is designated to each data in IP. After each TTM, the importance factor is decayed exponentially by using the forgetting rate (i.e., the new coming valid data have an importance factor equal to 1, and it is reduced for the next TTM period). The data in IP with a high importance factor are chosen as the active set IPactive . The SFCM + ANFIS method is then applied to IPactive to obtain a new fuzzy inference system (FIS). If the new FIS outperforms the old FIS, it will be substituted for the old FIS. This process will repeat for the next TTM if the entropy level of Channel 3 meets the described criteria (Fig. 3). It should be noted that there is a tradeoff for selecting importance factor. If the forgetting rate is high—which means the system forgets its background (dynamics) easily—then the system cannot cope with the situations in which the inputs have large dynamics. On the other hand, the forgetting rate should not be selected too low, which conducts a higher size of recorded data and larger storage space. In experiments here, the forgetting factors were set 2  0:4, which means the important factor of an input-output pair is near zero after 12 TTMs. 290 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 3, NO. 3, JULY-SEPTEMBER 2012 Fig. 2. Algorithm 1. Fig. 3. Flowchart for the control unit online training. The initial training phase is performed only once on the offline data for obtaining the initial control unit. REZAZADEH ET AL.: CO-ADAPTIVE AND AFFECTIVE HUMAN-MACHINE INTERFACE FOR IMPROVING TRAINING PERFORMANCES OF... 291 Fig. 4. Snapshot of virtual forearm within virtual environment. 2.8 Designing Virtual Training Environment An in-house virtual model composed of three rigid bodies— the upper arm, forearm, and hand—was built using MAYA software [32] based on Denavit-Hartenberg parameters and Leva’s study [11]. The virtual forearm is able to mimic forearm extension, flexion, supination, and pronation based on its input commands. Furthermore, a virtual ball and basket were also created and placed within the virtual environment, with their coordinates within the virtual environment randomly set (Fig. 4). When the coordinates of the end point of the virtual forearm equal the ball’s coordinates, the ball will be attached to the hand. Furthermore, when the hand that carries the attached ball reaches the coordinates of the basket, the ball will be released from the hand. The virtual forearm movement can be tuned to three different speed levels (slow, normal, and fast). A slow speed level can emulate a heavy prosthesis or load lifting task. A fast speed level can emulate a light prosthesis or a situation where the amputee wants to perform exercises to increase the dexterity of the upper arm muscles without wearing the prosthesis. 2.9 Experimental Protocol Prior to the online experiment, each participant was required to read through prepared training materials. Each participant was asked to use two different interfaces for controlling the virtual forearm: ACE-off and ACE-on. In the ACE-off interface, the control unit was not in the active mode, which means the inference system was not modified and updated according to Algorithm I. On the other hand, in the ACE-on interface, the control unit was in the active mode. In addition, to remove the influence of the learning process as much as possible, the sequence of using the interfaces was counter-balanced and the time interval between the two experiments was set at two weeks. 2.9.1 Online Recording Protocol The user was asked to participate in an online 60-minute experiment protocol. Meanwhile, the bioelectric signals from all of the data channels were recorded and processed as described in the above sections. The 60-minute experimental scenario was as follows: . . . Moving the virtual forearm end point to the coordinates of a ball. The ball will be attached to the virtual hand if it stays at the ball’s coordinates for 2 s. Moving the virtual forearm accompanied by the ball to the position of the basket. Fig. 5. Amputee (top and bottom left photos) and healthy subjects (bottom right) during task performance. Release the ball by staying at the basket’s position for 2 s. Each time the user performed the above task properly and gained a positive score, the corresponding completion time was simultaneously recorded. Then, the ball and basket positions were set randomly within the virtual environment for the next trial. It should be noted that no visual or auditory feedback about the obtained scores and completion time was provided. The 60-minute experimental period was divided into three 20-minute time slots and each time slot had a different forearm movement speed. The speed levels were set in the following order for a complete 60-minute experimental period: normal, slow, and fast. Fig. 5 shows the healthy and amputee users performing the experiments. . 2.9.2 Performance Metrics The overall quality of the interface was determined in terms of the subject’s physical, affective, and self-report metrics: Direct physical metrics. Score: This measure showed how many times the scenario was completed successfully by the user. According to [48], this measure can reflect the effectiveness of the user interface. . Completion time: The average time from when the user began the experimental scenario to when the scenario was completed. This measure mirrors the user interface efficiency [48]. . The average slope (changing rate) of the EMG entropy from Channels 1 and 2: These measures were calculated from the EMG signals recorded from Channels 1 and 2 to investigate whether there was any relation between the EMG entropy, alpha band entropy, and obtained score. Mental workload metrics. In the HMI area, a major goal of work psychology is the analysis of task demands in order to design interfaces and interaction schemes that bring about a lower mental workload and higher efficiency and effectiveness. This in turn will lead to lower stress levels and a decrease in the likelihood of errors. . 292 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 3, NO. 3, JULY-SEPTEMBER 2012 TABLE 1 Objective Performance Metrics Achieved by the Healthy Subjects under the ACE-On and ACE-Off States . . 3 Affective metric—The Average Slope of Alpha Subband from Channel 3: As described in Section 1.3, this objective measure can mirror the changes in mental workload while performing a task. Self-report metric: In this study, the NASA Task Load Index [48, 49] was employed. The NASA TLX has six metrics to assess mental workload (mental demand, physical demand, temporal demand, performance, effort, and frustration), and it was used to see whether the users really liked the dynamically adapting HMI and would be more likely to use it. A score from 0 to 100 was assigned to each metric after each of the experimental time slots (speed levels). This evaluation was used for validating the obtained objective physical and affective metrics. EXPERIMENTAL RESULTS AND ANALYSIS 3.1 Evaluating Effect of Proposed Control Scheme Table 1 shows the objective performance metrics achieved by the healthy subjects under two different experimental conditions: ACE-on and ACE-off. It is clear that by using the affective measures to update the inference system, the subjects achieved higher scores in the ACE-on (p-value < 0:001). In addition, the score and completion time were highly correlated with each other (correlation coefficient ¼ r ¼ 0:93; p-value < 0:001) because the score increased in a case where the subject could complete more tasks in a shorter time. Furthermore, the obtained score and completion time depended on the degree of muscular fatigue imposed by the interface. In a fatigued muscle, the fibers fire in a more synchronized way to compensate for the loss of muscle strength and exert the force adequate to handle the task. Therefore, the entropy in the fatigued muscle decreases. Table 1 shows that the slope of EMG entropy reduction in the biceps and triceps muscles are about 15 percent (average of all time slots) lower in ACE-on in comparison with the ACE-off state. This means that by using the affective control scheme, the degree of muscular fatigue will be reduced. As shown in Table 1, the slope of the EEG alpha range entropy remained in the same range by using the affective control scheme. However, in the ACE-off state, this slope increased as the level of difficulty increased. The alpha range entropy reduction (or retention) means the subject’s brain worked in a more organized and less complex way (see Section 1.3). In addition, the alpha band entropy is negatively correlated with the obtained score during the experiment (r ¼ 0:83; p-value < 0:05). Furthermore, despite the increasing slope of the EMG entropy during the experiment, the slope of the alpha range entropy remained the same as the ACE-on status. However, this phenomenon did not occur when the affective control scheme had the inactive status. In this case, the slope of the alpha band increased simultaneously with muscular fatigue. Table 2 shows the performance metrics for subject D1. The same trend as found in Table 1 can be seen in this table. The performance metrics for the ACE-on status are better REZAZADEH ET AL.: CO-ADAPTIVE AND AFFECTIVE HUMAN-MACHINE INTERFACE FOR IMPROVING TRAINING PERFORMANCES OF... 293 TABLE 2 Performance Metrics for Subject D1 than those for the ACE-off status. The results show that leaving the affective control scheme leads to an increase in the mental load during the experiment period. Therefore, the user had to exert more mental and physical effort to control and manipulate the virtual forearm, which led to muscular fatigue and a low score. Fig. 6 shows the Alpha band entropy and EMG entropy from Channels 1 and 2 within three TTM periods (20 minute) for subject D1 in the normal speed level and ACE-on state. The slope of the Alpha band entropy was higher at the beginning of the experiment, and it decreased after time passed. Moreover, the obtained score increased from 98 to 104 during this period, which means the subject gained more skill with the task and VRE. In addition, when evaluating the other subjects, similar results were achieved. According to the feedback from the questionnaires, we obtained the results above. This is probably because the participant was not relaxed but was nervous at the beginning of the experiment since he was unfamiliar with the experimental conditions. However, as time passed, the participant felt more relaxed as he adapted to the experiment. It should be noted that this condition could be experienced in the daily life of a prosthesis user who is trying to use and adapt to a real prosthesis. 3.2 Evaluating Robustness of Proposed Control Scheme To evaluate the robustness of the proposed classifier, each of the participants was asked to perform a protocol similar to that in Section 2.3 right after the 60-minute experimental period. The discrimination ratios for the classifier were obtained according to Table 3. It shows that for the healthy subjects, the average discriminating power of the stagnant classifier dropped 13.3 percent compared to a 2.2 percent drop for the adaptable classifier. Similar results were achieved for subject D1, where the discrimination ratio reductions were 15.9 and 1.6 percent for the nonadaptable and adaptable classifiers, respectively. This is a very important outcome which shows the robustness of the Fig. 6. Affective and physical performance measures from subject D1 during normal speed level task. 294 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 3, NO. 3, JULY-SEPTEMBER 2012 TABLE 3 Average Discrimination Rations of the Classifier at the Start and End of the Experiment * Discrimination ratio = number of correct discriminated movements/total number of movements. proposed control scheme despite variations in the myoelectric signals over the whole experiment period. It should be noted that subject D1 had some minor difficulties at the beginning in generating two different commands. Thus, the discriminating ratios of the classifier were low in comparison with the healthy subjects. 3.3 Evaluating Mental Workload and Usability of Proposed Method A conclusion about the performance of the proposed control scheme could not be reached unless the subjects’ self-reports correlated with the objective performance metrics. Thus, the self-report ratings of subjects were completed to see whether the users really liked the dynamically adapting version more and would be more likely to use it (allowing for the fact that they must wear additional sensors, of course). Table 4 shows the results of the completed questionnaires for all of the participants. It is clear that the self-report metrics in the ACEon condition outperformed the ACE-off for all of the time slots (p-value < 0:001). Table 4 also reveals that for the slow speed time slot, the self-report bands for the performance, temporal, mental, and physical demands are lower than the other time slots. Thus, this slot of the experiment was more difficult than the other time slots for all of the subjects. Furthermore, the subjects were asked whether the proposed control scheme decreased the HMI usability by TABLE 4 Results of the Completed Questionnaires for All of the Participants * p-value < 0:001. REZAZADEH ET AL.: CO-ADAPTIVE AND AFFECTIVE HUMAN-MACHINE INTERFACE FOR IMPROVING TRAINING PERFORMANCES OF... adding additional forehead electrodes. Only one healthy subject complained about attaching these electrodes to his forehead. However, the rest of the participants (including D1) preferred to wear this electrode pair to gain better performance results. The Pearson correlation test was conducted on the experimental results to explore whether there was any correlation between the self-report metrics and objective performance metrics. The results show that the self-report performance metrics and the obtained scores were highly correlated (r > 0:9; p-value < 0:01). The same trends were also found for the task temporal demands and task completion time (r > 0:9; p-value < 0:01), task physical demands and muscular fatigue (r > 0:84; p-value < 0:05), and the task mental demand and slope of the alpha band entropy (r > 0:9; p-value < 0:01). Thus, the results reveal that the proposed user interface enhances both the subjective and objective performance metrics. 4 DISCUSSION To explore the effect of the affective control scheme on the subjective and objective performance metrics, the abovementioned experiments were performed using 16 subjects based on the described protocol. It was found that a high correlation between the subjective and objective measures exists. When the control scheme was in the active state (ACEon), the direct physical and affective measures were superior to those when the control scheme was inactive (ACE-off). It was found that the increasing rate for the alpha subband entropy was slow with ACE-on. This indicates that despite changes in the nature of the EMG signal caused by physical fatigue, the subject could still handle the requested tasks. The completed questionnaires and obtained scores emphasized this fact. According to the obtained results, the proposed method can retain or decrease the instantaneous and cumulative mental workload, prevent the reduction of temporal stability, and increase the interface’s usability. As a result, the user (his brain and muscles) achieves more skill and dexterity in the experimental period, which could prevent muscle fatigue because untrained muscle is highly fatigable and shows faster contractile speeds [36]. Because, in the ACE-off state, the control scheme could not be adapted to the user’s physical and emotional changes over time, the subject’s mental workload was higher in comparison to the ACE-on state. The experimental results for subject D1 also showed that using the proposed method has benefits over conventional systems (without affective feedback). Furthermore, the outcomes could be used as a proxy for the potential application for users with disabilities. The proposed system was tested for different monitoring periods (TTM) in order to explore a suitable range of pacing time for adaptation. To cope with variability in the environment, TTM should be within the intermediate range. However, as a general guideline, the TTM period should be selected according to context changes over time (the dynamics of the environment), the experimental protocol, and the maximum admissible accumulated delay. In a nonadaptable control scheme, the performance of the system is only related to the classifier’s accuracy. Thus, 295 to cope with input signal variations, designing a control algorithm with complex classification methods is inevitable. However, in our control scheme, the system retains its robustness and accuracy while providing more usability for its user by employing a simple heuristic, rather than a complex processing algorithm. Moreover, it reduces the need for subject retraining or data recollection. It should be mentioned that in any interface the user continuously tries to adapt to changes in the working space workload while performing the requested task. In the proposed method, the designed controller employs the physiological reaction of the user’s behavior to reduce the mental workload. Thus, there are two adaptive subsystems that work in a counterproductive way [35]. 5 CONCLUSION AND FUTURE WORK This paper presents a real-time and adaptable human machine interface for controlling a virtual forearm based on the user’s affective states. The proposed HMI attempts to adapt itself to its user’s affective status while the user is trying to undertake the experiment. It is a mixed initiative adaptation by which the control unit collaborates with the user. The machine changes the user’s alpha subband complexity by providing modifying itself to meet the metal and physical states of a user. In addition, the user modifies the classifier’s inference system by Algorithm 1, in which the updating criteria is selected by the user’s alpha band entropy. Thus, this bidirectional modification (co-adaptation) brings some advantages, such as: 1. 2. 3. Enhancing the system’s ecology and context awareness: The extracted mental states are correlated with the subject’s emotional status and the physical performance metrics (obtained score, completion time) during the experimental period. Thus, the extracted cues can reveal the user’s emotion about the task (affective cues). By passing the affective cues to the interface, the interface’s awareness of the user’s conditions (mental and physical) is enhanced. The user is also aware of the virtual forearm status through the visual feedback. The visualization of the movement may increase the perceptual brain process, facilitate the motor activity, and may optimally induce cortical plasticity [50], [51], [52] and functional recovery in disabled patients. The bidirectional information flow within the interface improves the context awareness of the entire system (user, virtual forearm, and environment). Co-adaptive HMI: The proposed HMI can be adapted to its user’s affective states by employing the realtime heuristic and mixed initiative algorithm. In addition, the user’s adaptation to the virtual forehand can also be facilitated. Thus, the two parties are collaborating and interacting mutually. Increasing the usability: The classifier accuracy is increased or retained as a result of the intuitive interaction between the user and machine (control unit). Consequently, higher performance measures are achieved in comparison to a conventional control methodology. Thus, the user’s mental workload is 296 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, reduced, and therefore can retain control of the virtual prosthesis for a longer period of time. In addition, based on the obtained results and the completed questionnaires, it is clear that the proposed control scheme improves the functionality of the HMI by allowing the user to function effectively as a central part of the interface and decreasing the needed mental workload for performing the task. In future work, the research will be expanded to more complex virtual environments, different tasks for motor skill empowering, and more subjects with disabilities. In addition, an error-related potential (ERP) approach will be deployed in the control scheme to make the system more flexible and robust to cope with nonroutine tasks and situations. REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] D.G. Kamper, R L. Harvey, S. Suresh, and W.Z Rymer, “Relative Contributions of Neural Mechanisms Versus Muscle Mechanics in Promoting Finger Extension Deficits Following Stroke,” J. Muscle Nerve, vol. 28, pp. 309-318, 2003. D.G. Kamper and W.Z. Rymer, “Impairment of Voluntary Control of Finger Motion Following Stroke: Role of Inappropriate Muscle Co-Activation,” J. Muscle Nerve, vol. 24, pp. 673-681, 2001. K. Eng, E. Siekierka, P. Pyk, E. Chevrier, Y. Hauser, M. Cameirao, L. Holper, K. Hägni, L. Zimmerli, A. Duff, C. Schuster, C. Bassetti, P. Verschure, and D. Kiper, “Interactive Visuo-Motor Therapy for Stroke Rehabilitation,” J. Medical, Biological Eng. and Computing, vol. 45, pp. 901-907, 2007. M. Kelly-Hayes, J.T. Robertson, J.P. Bronderick, P.W. Duncan, L.A. Hershey, E.J. Roth, W.H. Thies, and C.A. Trombly, “The American Heart Association Stroke Outcome Classification,” J. Stroke, vol. 29, pp. 1274-1280, 1998. D. Jack, R. Boian, A.S. Merians, M. Tremaine, G.C. Burdea, S.V. Adamovich, M. Recce, and M.H. Poizner, “Virtual Reality-Enhanced Stroke Rehabilitation,” IEEE Trans. Neural Systems and Rehabilitation Eng., vol. 9, no. 3, pp. 308-318, Sept. 2001. M. Zecca, J. Carpaneto, S. Micera, M.C. Carrozza, P. Dario, K. Itoh, and A. Takanishi, “Evolutionary Design of a Fuzzy Classifier for EMG-Based Control—Control of a Multi-DoFs Underactuated Hand Prosthesis,” Proc. Robotics and Mechatronics Conf., pp. 26-28, May 2006. D.H. Silcox, M.D. Rooks, R.R. Vogel, and L.L. Fleming, “Myoelectric Protheses,” The J. Joint and Bone Surgery, vol. 75, no. 12, pp. 1781-1791, 1993. P. Shenoy, K.J. Miller, B. Crawford, and R.N. Rao, “Online Electromyographic Control of a Robotic Prosthesis,” IEEE Trans. Biomedical Eng. vol. 55, no. 3, pp. 1128-1135, Mar. 2008. M. Zecca, S. Micera, M.C. Carrozza, and P. Dario, “On the Control of Multifunctional Prosthetic Hands by Processing the Electromyographic Signal,” Critical Rev. in Biomedical Eng., vol. 30, nos. 46, pp. 459-485, 2002. M.A. Oskoei and H. Hu, “Myoelectric Control Systems—A Survey,” J. Biomedical Signal Processing and Control, vol. 2, no. 4, pp. 275-294, 2007. J.L. Pons, Wearable Robots: Biomechatronic Exoskeleton, first ed., pp. 1-16. Wiley & Sons, 2008. R. Kato, T. Fujita, H. Yokoi, and T. Arai, “Adaptable EMG Prosthetic Hand Using On-Line Learning Method-Investigation of Mutual Adaptation between Human and Adaptable Machine,” Proc. IEEE 15th Int’l Symp. Robot and Human Interactive Comm., pp. 599-604, Sept. 2006. K. Momen, S. Krishnan, and T. Chau, “Real-Time Classification of Forearm Electromyographic Signals Corresponding to UserSelected Intentional Movements for Multifunction Control,” IEEE Trans. Neural Systems and Rehabilitation Eng., vol. 15, no. 4, pp. 535542, Dec. 2007. A.B. Ajiboye and R.F. Weir, “A Heuristic Fuzzy Logic Approach to EMG Pattern Recognition for Multi-functional Prosthesis Control,” IEEE Trans. Neural Systems and Rehabilitation Eng., vol. 13, no. 3, pp. 280-291, Sept. 2005. VOL. 3, NO. 3, JULY-SEPTEMBER 2012 [15] D. Nishikawa, “On-Line Learning Based Electromyogram to Forearm Motion Classifier with Motor Skill Evaluation,” JSME Int’l J. Series C, vol. 43, no. 4, pp. 906-915, 2000. [16] O. Fukuda, T. Tsuji, M. Kaneko, and A. Otsuka, “A HumanAssisting Manipulator Tele-Operated by EMG Signals and Arm Motions,” IEEE Trans. Robotics and Automation, vol. 19, no. 2, pp. 210-222, Apr. 2003. [17] M. Holden, “Virtual Environments for Motor Rehabilitation: Review,” J. Cyberpsychology and Behavior, vol. 8, no. 3, pp. 187211, 2005. [18] I. Mohammad Rezazadeh, X. Wang, M. Firoozabadi, and M. Hashemi Golpayegani, “Using Affective Human-Machine Interface to Increase the Operation Performance in Virtual Construction Crane Training System: A Novel Approach Automation in Construction,” Automation in Construction, vol. 20, pp. 289-298, 2011. [19] I. Mohammad Rezazadeh, X. Wang, R. Wang, and S.M.P. Firoozabadi, “Toward Affective Handsfree Human-Machine Interface Approach in Virtual Environment-Based Equipment Operation Training,” Proc. Ninth Int’l Conf. Construction Applications of Virtual Reality, 2009. [20] E.P. Doherty, G. Cockton, J. Rizzo, B. Blondina, and B. Davis, “Yes/No or Maybe—Further Evaluation of an Interface for BrainInjured Individuals,” Interacting with Computers, vol. 14, no. 4, pp. 341-358, 2002. [21] T. Surdilovic and Y-Q Zhang, “Convenient Intelligent Cursor Control Web Systems for Internet Users with Severe MotorImpairments,” Int’l J. Medical Informatics, vol. 75, no. 1, pp. 86-100, 2006. [22] Cyberlink, www.brainfinger.com, Mar. 2011. [23] Biopac, http://www.biopac.com, Mar. 2011. [24] V. Moertini, “Introduction to Five Data Clustering Algorithm,” Integral, vol. 7, no. 2, 2002. [25] A. Priyona, M. Ridwan, A. Alias, R. Atiq, R. Rahmat, A. Hassan, and M. Ali, “Generation of Fuzzy Rules with Subtractive Clustering,” Universiti Teknologi Malaysia, Jurnal Teknologi, vol. 43(D) Dis, pp. 143-153, 2003. [26] K. Polat, S. Yosunkaya, and S. Gunes, “Comparison of Different Classifier Algorithms on the Automated Detection of Obstructive Sleep Apnea Syndrome,” J. Medical Systems, vol. 32, pp. 243-250, 2008. [27] M. Khezri and M. Jahed, “Real-Time Intelligent Pattern Recognition Algorithm for Surface EMG Signals,” Biomedical Eng. Online, vol. 3, no. 6, p. 45, 2003. [28] N.J. ovec, K.J. ovec, and I. Gerlic, “The Influence of Mozart’s Music on Brain Activity in the Process of Learning,” Clinical Neurophysiology, vol. 117, pp. 2703-2714, 2006. [29] S. Micheloyannis, M. Vourkas, M. Bizas, P. Simos, and C.J. Stam, “Changes in Linear and Nonlinear EEG Measures as a Function of Task Complexity: Evidence for Local and Distant Signal Synchronization,” J. Brain Topography, vol. 15, no. 4, pp. 239-247, Summer 2003. [30] J. Hu, D. Xiao, and Z. Mu, “Application of Energy Entropy in Motor Imagery EEG Classification,” Int’l J. Digital Content Technology and Its Appl., vol. 3, no. 2, pp. 83-90, June 2009. [31] S. Aydin, H.M. Saraoglu, and S. Kara, “Log Energy Entropy-Based EEG Classification with Multilayer Neural Networks in Seizure,” Annals of Biomedical Eng., vol. 37, no. 12, pp. 2626-2630, Dec. 2009. [32] MAYA, www.autodesc.com/maya, May 2010. [33] Mathworks, www.mathworks.com, May 2010. [34] R. Hornero, D.E. Abasolo, N. Jimeno, and P. Espino, “Applying Approximate Entropy and Central Tendency Measure to Analyze Time Series Generated by Schizophrenic Patients,” Proc. 25th Ann. IEEE Int’l Conf. Eng. in Medicine and Biology Soc., Sept. 2003. [35] J. Liu, C. Zhang, and C. Zheng, “EEG-Based Estimation of Mental Fatigue by Using KPCA-HMM and Complexity Parameters,” J. Biomedical Signal Processing and Control, vol. 5, pp. 124-130, 2010. [36] L.A. Frey Law and R.K. Shields, “Mathematical Models of Human Paralyzed Muscle After Long-Term Training,” J. Biomechanics, vol. 40, pp. 2587-2595, 2007. [37] M. Serruyaa and M.J. Kahanab, “Techniques and Devices to Restore Cognition,” Behavioural Brain Research, vol. 192, pp. 149165, 2008. [38] Q.H. Mach, M.D. Hunter, and R.S. Grewal, “Neurophysiological Correlates in Interface Design: An HCI Perspective,” Computers in Human Behavior, vol. 26, pp. 371-376, 2010. REZAZADEH ET AL.: CO-ADAPTIVE AND AFFECTIVE HUMAN-MACHINE INTERFACE FOR IMPROVING TRAINING PERFORMANCES OF... [39] B.H. Kantowitz, “Mental Workload,” Human Factors Psychology, P.A. Hancock ed., pp. 81-121, Elsevier, 1987. [40] J.A. Coan and J.J. Allen, “Frontal EEG Asymmetry as a Moderator and Mediator of Emotion,” Biological Psychology, vol. 67, nos. 1/2, pp. 7-49, 2004. [41] H. Matsunaga and H. Nakazawa, “Design Method Consideration Human Satisfaction: Development of Adaptive Human-Machine Interface Based on Satisfaction Measures,” Human Factors and Ergonomics in Manufacturing, vol 9, no. 3, pp. 253-266, 1999. [42] M.J. Larson, D.A. Good, and J.E. Fair, “The Relationship between Performance Monitoring, Satisfaction with Life, and Positive Personality Traits,” Biological Psychology, vol. 83, pp. 222-228, 2010. [43] J. Sleigh, L. Voss, and J. Barnard, “What Are Electroencephalogram Entropies Really Measuring?” Int’l Congress Series, vol. 1283, pp. 231-234, 2005. [44] O.A. Rosso, “Entropy Changes in Brain Function,” Int’l J. Psychophysiology, vol. 64, pp. 75-80, 2007. [45] H. Yoo, M. Kim, and O. Kwon, “Emotional Index Measurement Method for Context-Aware Service,” Expert Systems with Applications, vol. 38, no. 1, pp. 785-793, 2010. [46] J. Coan, J.J. Allen, and E. Harmon-Jones, “Voluntary Facial Expression and Hemispheric Asymmetry over the Frontal Cortex,” Psychophysiology, vol. 38, pp. 912-925, 2010. [47] T. Liu, J. Shi, D. Zhao, and J. Yang, “The Relationship between EEG Band Power, Cognitive Processing and Intelligence in SchoolAge Children,” Psychology Science Quarterly, vol. 50, no. 2, pp. 259268, 2008. [48] C.S. Nam, S. Johnson, Y. Li, and Y. Seong, “Evaluation of HumanAgent User Interfaces in Multi-Agent Systems,” Int’l J. Industrial Ergonomics, vol. 39, pp. 192-201, 2009. [49] S. Rubio, E. Diaz, and J. Martin, “Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods,” Applied Psychology: An Int’l Rev., vol. 53, pp. 6186, 2004. [50] D. Perani, F. Fazio, N.A. Borghese, M. Tettamanti, S. Ferrari, J. Decety, and M.C. Gilardi, “Different Brain Correlates for Watching Real and Virtual Hand Actions,” NeuroImage, vol. 14, pp. 749758, 2001. [51] S. Subramanian, L.A. Knaut, C. Beaudoin, B.J. McFadyen, A.G. Feldman, and M.F. Levin, “Virtual Reality Environments for PostStroke Arm Rehabilitation,” J. NeuroEng. and Rehabilitation, vol. 4, no. 2, p. 20, 2007. [52] H. Sveistrup, “Motor Rehabilitation Using Virtual Reality,” J. NeuroEng. and Rehabilitation, vol. 1, no. 1, p. 10, 2004. [53] C. Berka, “EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks,” Aviation, Space, and Environmental Medicine, vol. 78, no. 5, pp. B231-B244, May 2007. [54] A.W.K. Gaillard, “Stress, Workload, and Fatigue as Three Biobehavioural States: A General Overview,” Stress, Workload, and Fatigue, P.A. Hancock and P.A. Desmond eds., pp. 623-639, Erlbaum, 2001. [55] M. Wyczesany1, J. Kaiser1, and A.M.L. Coenen, “Subjective Mood Estimation Co-Varies with Spectral Power EEG Characteristics,” Acta Neurobiologiae Experimentalis, vol. 68, pp. 180-192, 2008. [56] G. Chanel, C. Rebetez, M. Betrancourt, and T. Pun, “Emotion Assessment from Physiological Signals for Adaptation of Games Difficulty,” IEEE Trans. Systems, Man, and Cybernetics—Part A: Systems and Humans, vol. 41, pp. 1052-1063, Nov. 2011. [57] G. Chanel, C. Rebetez, M. Betrancourt, and T. Pun, “Boredom, Engagement and Anxiety as Indicators for Adaptation to Difficulty in Games,” Proc. 12th Int’l Conf. Entertainment and Media in the Ubiquitous Era, Oct. 2008. [58] SENIAM Project Surface ElectroMyoGraphy for the Non-Invasive Assessment of Muscles, http://www.seniam.org/, Nov. 2011. View publication stats 297 Iman Mohammad Rezazadeh received the PhD degree in biomedical engineering from Science and Research Branch, Islamic Azad University (SRIAU), Tehran, Iran in 2011. He is now a postdoctoral scholar at the Center for Mind and Brain of the University of California, Davis (UCD) and working on multisensory integration in brain during its stimulation. He is also a faculty member and researcher in the Department of Biomedical Engineering in SRIAU. His current research interests include designing co-adaptive human-machine interfaces, understanding brain complexity, emotional intelligence and affective computing, cyber collaboration, and virtual reality. Mohammad Firoozabadi received the BSc degree in electronics engineering from the University of Tabriz, in 1987, the MSc degree in electronics engineering from Amir-Kabir University of Technology, Tehran, in 1991, and the PhD degree in electrical engineering (biomedical engineering) from Tarbiat Modares University, Tehran, Iran, in 1997. His research and teaching interests include theory and application of bioelectromagnetics, human-machine interaction, bioelectric phenomena and electrophysiology, bioinstrumentation, and biological signal processing. He serves as a professor of biomedical engineering and the head of the Biomedical Informatics Department at Tarbiat Modares University, and as the president of the Iranian Society of Biomedical Engineering. He has published more than 260 papers in peer reviewed journals and international conferences. He was a research deputy dean on the Medical Sciences Faculty from 20092011, the manager of the Apllied Research Office (1997-2005), and the head of the Medical Physics Department (2008-2010) at Tarbiat Modares University. He has received many awards, including the Razie second rank prize of “Invention, Innovation, Novation Group” from the Iranian National Research Center of Medical Sciences (NRCMS) for “design and implementation of isolated heart pacing, data logger, and the signals analyzer system” in 2004. Huosheng Hu is a professor in the School of Computer Science and Electronic Engineering at the University of Essex, United Kingdom, leading the Robotics Research Group. His research interests include behavior-based robotics, human-robot interaction, service robots, embedded systems, data fusion, learning algorithms, mechatronics, and pervasive computing. He has published more than 360 papers in journals, books, and conference proceedings in these areas, and received a number of best paper awards. He has been a program chair or a member of the advisory/organizing committee for many international conferences such as IEEE ICRA, IROS, ICMA, ROBIO, ICIA, ICAL, and IASTED RA, CA, CI conferences. He currently serves as the editor-in-chief for the International Journal of Automation and Computing and as executive editor for the International Journal of Mechatronics and Automation. He is a founding member of the IEEE Robotics & Automation Society Technical Committee on Networked Robots, a fellow of IET and the InstMC, and a senior member of the IEEE and ACM. S. Mohammad Reza Hashemi Golpayegani received the BS degree in electrical engineering from Amir-Kabir University of Technology, Tehran, in 1968, the MS degree in electrical engineering from the University of Dayton in 1973, and the PhD degree in biomedical engineering from the Ohio State University in 1976. He has published more than 70 scientific papers in journals, books, and conference proceedings in the areas of control system theory, biomedical engineering, and chaos. He is a full professor of biomedical engineering at Amir-Kabir University of Technology. He has been recognized with the most prestigious rank for outstanding Iranian professors—the long-lasting scientific faces “Chehreyeh Mandegar”— in 2005. His current research interests include nonlinear dynamics and chaos, cybernetic, and complex systems.