Interacting with Computers 16 (2004) 857–878 www.elsevier.com/locate/intcom A research agenda for physiological computing Jennifer Allansona, Stephen H. Faircloughb,* a School of Computing and Mathematical Sciences, Liverpool John Moores University, Byrom Street, Liverpool L3 3AF, UK b School of Psychology, Liverpool John Moores University, Henry Cotton Campus, 15–21 Webster Street, Liverpool L3 2ET, UK Available online 6 October 2004 Abstract Physiological computing involves the direct interfacing of human physiology and computer technology, i.e. brain–computer interaction (BCI). The goal of physiological computing is to transform bioelectrical signals from the human nervous system into real-time computer input in order to enhance and enrich the interactive experience. Physiological computing has tremendous potential for interactive innovation but research activities are often disparate and uneven, and fail to reflect the multidisciplinary nature of the topic. This paper will provide a primer on detectable human physiology as an input source, a summary of relevant research and a research agenda to aid the future development of interactive systems that utilise physiological information. q 2004 Elsevier B.V. All rights reserved. Keywords: Physiological computing; Biofeedback; Brain–computer interaction (BCI); Affective computing 1. Introduction The human body is chemical, electrical, mechanical, thermal and magnetic in nature. Individual human senses are tuned to receive different kinds of data such as smell and taste (chemical), sound (mechanical), touch (mechanical, thermal) and sight (electromagnetic). The sensory organs of a computer correspond to the input devices it supports. Physiological sensors can be used to detect information such as heart rate and brain signals that reflect changes in a user’s mood and/or environment. The growing commercial * Corresponding author. Tel.: C44 151 2314199; fax: C44 151 2314245. E-mail addresses: [email protected] (J. Allanson), [email protected] (S.H. Fairclough). 0953-5438/$ - see front matter q 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.intcom.2004.08.001 858 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 availability of these sensing technologies enables the creation of physiological computers—interactive systems that take detectable human physiology as an input source. Physiological computers are poised to revolutionise HCI, ushering in a range of new and exciting modes of interaction. The aim of this paper is to explore the rich possibilities of physiological computing in a bid to set out a research agenda for the development of future physiological computing solutions. In Section 2, we provide a general introduction to detectable human physiology, illustrating in each case the utility of a given physiological parameter in HCI terms. Section 3 of the paper describes two categories of physiological computer—biofeedback-based systems and adaptive biocybernetic systems. These categorisations are based on a combination of passive monitoring versus active control of physiological signals and user feedback. They are more illustrative than definitive. Using these categories we will explore one particularly challenging physiological computing application, namely brain–computer interaction. Section 5 of the paper outlines a program of work to be carried out in order to realise the full potential of physiologically interactive computer systems. 2. Detectable human physiology The earliest means of detecting the subtle indicators of physiological functioning was direct observation, i.e. an ear placed on the chest to hear the rhythmical beating of the heart. Specialised mechanical and electrical devices to amplify physiological information began to appear in the early 20th century and as computers became popular, so peripheral physiological sensors were developed to utilise their data processing and display capabilities. Psychophysiologists have been studying detectable physiological signals for over 70 years in a bid to understand the body’s responses to changing psychological and physical conditions. HCI practitioners currently use physiological indicators as usability metrics and as an input source for affective/emotional computing. Contemporary physiological sensors detect activity from three areas of the human nervous system: the Central Nervous System (CNS), the Somatic Nervous System (SNS) and the Autonomic Nervous System (ANS). The CNS includes the brain and the spinal cord, the SNS is concerned with the control of muscles, and the ANS controls and coordinates the major glands and organs of the body. In the following sections we will describe a range of measures that can be taken from the CNS, SNS and ANS and give examples of how each can be incorporated into physiological computing applications. 2.1. The electroencephalogram (EEG) Electrodes placed on the scalp measure the electrical activity of the cortex or surface of the brain. EEG can be used to monitor the state of alertness of the waking brain and is a useful usability metric, especially in the evaluation of safety critical systems. For example, high frequency ‘beta’ activity (14–30 Hz) is associated with a state of high alertness, ‘alpha’ activity (8–12 Hz) indicates a state of relaxed alertness and ‘delta’ activity (0.5–3.5 Hz) is associated with sleepiness. By monitoring EEG while a user completes J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 859 a task it is possible to measure task engagement (Pope et al., 1995) or task difficulty and to identify lapses in the attention (Mekeig and Inlow, 1993). Recent research has demonstrated that ‘theta’ activity (4–7 Hz) from frontal sites on the scalp increases as a task becomes more difficult or demanding. Conversely, alpha activity tends to become suppressed if task difficulty rises (Gevins and Smith, 2003; Gevins et al., 1998; Wilson et al., 1999). As well as monitoring continuous, oscillating EEG signals we can also look for discrete, repeatable electrical responses in the brain to particular stimuli. These responses, known as event-related potentials (ERPs), appear in direct relation to either discrete psychological events (Kramer, 1987), flashing stimuli (Farewell and Donchin, 1988) or action preparation, i.e. thinking about moving the foot or the hand (Pfurtscheller et al., 1997). These discrete, repeatable signals have been demonstrated as reliable input control signals for hands-free control applications (discussed further in Section 3.2). 2.2. The electromyogram (EMG) By positioning two electrodes on the skin over an appropriate muscle, it is possible to record electrical activity that serves as a correlate of the kinaesthetic activity of the muscle. The resulting signal can be used to deduce whether the state of the muscle is complete contraction, partial contraction or complete relaxation. As it indicates the subtlest signs of muscle activity, EMG is a good indicator of motor preparation for movement (Malmo and Malmo, 2000). As increasing muscle tension can be indicative of negative affect EMG has been explored as a potential metric to assess users’ sense of presence in virtual environments (Weiderhold et al., 2003). Psychophysiological research examining EMG has demonstrated consistent changes in facial EMG in response to pleasant and unpleasant stimuli, particularly in the muscle structure of the eyebrow (Cacioppo et al., 1990). This ability to monitor the expressiveness of the forehead provides a useful usability metric and also represents a potential input for emotionally responsive or affective computer systems. 2.3. The electro-oculogram (EOG) and pupillometry The electro-oculogram is a measure of the potential difference between the cornea and the retina of the eye. Measurement of the EOG through electrodes positioned around the eye provides information about both the changing position and speed of movement of the eye. The use of EOG to localise the x–y coordinates of the eye is dependent on the static position of the head, which is the reason why video-based eye-tracking is more popular. But whichever the approach used, eye movement capture can provide important cues about the allocation of visual attention to different components of the user interface and act as input for the design of Attentive User Interfaces (Vertegaal, 2003). Video-based eye tracking devices are a good way to measure the pupillary response. Circular and radial muscle fibres control both constriction and dilation of the pupil. Pupil size responds to emotional stimuli, whether it is positive or negative in tone (Partala and Surakka, 2003) providing another potentially useful input for affective computing applications. Changes in pupil size may also be time-indexed to the presentation of 860 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 psychological stimuli. This type of measurement yields a task-evoked pupillary response which reflects the level of cognitive demand induced by the stimuli (Beatty, 1982) and is thus applicable to usability testing. EOG sensors can also be used to measure eye blinks. The measurement of blink rate and duration yield meaningful information about task demand (Yamada, 1998) and levels of fatigue (Stern et al., 1994), making it a good metric for interface evaluation and usability testing. In addition an eye-blink can be used as switching signal, equivalent to a button click, for a hands-free mouse. 2.4. The electrocardiogram (ECG) The electrocardiogram is a measure of electrical events associated with contraction of the heart muscle. The measurement of heart rate (HR)—the speed at which the heart is beating—is usually expressed in beats-per-minute (bpm). Both psychological stimuli and physical activity (Porges and Byrne, 1992) increase heart rate. For example, if an interface task is challenging due to cognitive demands, time restrictions or uncertainty, HR will generally increase (Boutcher et al., 1998; Carroll et al., 1986a,b). HR has been used previously to evaluate the stimulating effects of computer games (Calvert and Tan, 1994). More recently, it has been incorporated into computer games that alter the level of challenge in real time based on detected changes in a player’s heart rate (Gilleade and Allanson, 2003). Besides psychological processes like challenge, a range of bodily activities influences the heart. The control of the heart rate represents an amalgamation of thermoregulation, blood pressure control and the influence of respiratory patterns. Researchers have attempted to distil these three influences by subjecting the heart period data to a mathematical technique called Fast Fourier Transform (FFT) (Aasman et al., 1987). This research has identified a mid-frequency component known as 0.1 Hz sinus arrhythmia (or mid-frequency component of heart rate variability HRV), which has been identified with mental effort in response to both laboratory tasks (Mulder, 1986) and real-life activities (Tattersall and Hockey, 1995). The heart rate variability measure is reviewed in detail by Berntson et al. (1997). 2.5. Respiratory patterns The diaphragm of the chest expands and contracts as a person inhales and exhales. The rate and depth of respiratory activity may be measured with a band sensor secured around the chest. Respiratory rate may be quantified as the number of breaths per minute. In terms of respiration rate as a usability metric it is of note that breathing rate increases in relation to task demand (Backs and Selijos, 1994). Longer, deeper breaths occur when an interactive task is difficult or demanding (Veltman and Gaillard, 1998). It has also been claimed that respiratory patterns may reflect emotional dichotomies such as calmexcitement and relaxation-tension (Boiten, 1998) which may be useful for affective computing applications. In addition, voluntary respiration has previously been used as a hands-free interaction mechanism with virtual reality interfaces (Davies and Harrison, 1996). The interested reader is referred to the review of methodology by Wientjes (1992). J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 861 2.6. Electrodermal activity/galvanic skin response (GSR) If two electrodes are placed on the skin and a small constant current is driven through them, the skin can be seen to behave as a variable resistor. A voltage develops across the electrodes and application of Ohm’s law can be used to calculate the effective resistance of the skin. Electrodermal activity responds to emotional stimuli such as music, observed violence, erotic stimuli, etc. and as such lends itself to interface evaluation. This measure has also been implicated in the measurement of mental workload (Verwey and Veltman, 1996). The fingers, palm, forearm and the soles of the feet are active sites for the measurement of electrodermal activity. Consequently, consideration must be given to the means of recording this type and how it might affect the normal movement/behaviour of the individual. 2.7. Blood pressure Blood pressure is a measurement of the force with which the heart is pumping blood around the body. A measurement of blood pressure also reflects the resistance characteristics of the arteries through which the blood is flowing. Resting levels of blood pressure (BP) are influenced by a variety of demographic factors such as personality (i.e. hostile individuals have higher levels of resting BP) and aerobic fitness. BP has a tendency to increase under conditions of active coping (Light, 1981). It has been suggested that patterns of ECG and BP may be used to differentiate between humans in a state of challenge and a state of threat (Blascovich et al., 1999). This information is particularly useful when considering the evaluation of critical systems interfaces and the design of computer games. What is not apparent from the preceding description of psychophysiology is the characteristic nature of the data. As an input source for computers, physiological signals are a far less reliable data source than we are used to. For a start each individual subject’s physiology (and consequently the signatory characteristics of their physiological activity) is unique to them. Potential ranges of ‘normal’ psychophysiological activity must take into consideration factors such as gender, age and general health. As if this were not problematical enough, most psychophysiological parameters are susceptible to environmental effects such as changes in temperature and humidity. Cardiac activity (heart rate/blood pressure) can also be influenced by factors such as the smoking, posture and the time of day (Siddle and Turpin, 1980). A comprehensive description of the issues surrounding physiological signal detection and evaluation can be found in Martin and Venables (1980). At first consideration, these variable factors may appear to preclude the utility of physiological information as an input source. However, work on context-aware and ubiquitous computing is providing information about the user’s environment that can be used to normalise anomalies in the physiological data stream. Of course, any truly useful physiological computer system will include the processing capabilities required to learn the response characteristics of a particular user over time and in different situations, an issue discussed in the research agenda presented later. The interested reader is referred to 862 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 the relevant chapters in texts by Andreassi (1995) and Cacioppo et al. (2000) for further information regarding the factors which influence psychophysiological indicators. 3. Categories of physiological computing A range of HCI applications, from affective computing, through usability evaluation to hands-free control, require systems that detect and process a user’s changing physiological state. For the purpose of this paper the applications of physiological computing have been divided into two categories: biofeedback-based systems and biocybernetically adaptative systems. This classification is based on: † the flow of information between psychophysiology and computer hardware † the type of computer processing required 3.1. Biofeedback In 1948, Wiener first published his treatise on communication and control in biological and mechanical systems (Wiener, 1961). One of the core principles of his cybernetic theory is that of feedback, where the controller of a system can control a given variable if information about that variable is made available to it. During the late 1960s work from several disciplines, including psychophysiology, behavioural medicine, stress research and consciousness research were diverging on the notion of humans being able to exert conscious influence over seemingly unconscious physiology. In many cases it was found that feeding back physiological information to a subject was the key to successful physiological control. This process, called biofeedback, is best summarised by Olson (1995, p. 29): “.a group of therapeutic procedures that.utilise electronic or electromechanical instruments.to accurately measure, process and feed back to persons and their therapists.information with educational and reinforcing properties.about neuromuscular and autonomic activity, both normal and abnormal.in the form of analogue or binary, auditory and/visual feedback signals.to help persons develop greater awareness of, confidence in and an increase in voluntary control over their physiological processes that are otherwise outside awareness and/or less voluntary control,.by first controlling the external signal.” The physiological signal feedback loop for a system incorporating computer-based signal presentation is shown in Fig. 1. The role of the computer is to retrieve physiological signals from the sensing hardware, pre-process the signals and display those signals back to subject in real time. At the present time computer-based physiological signal presentation systems are developed to service two distinct applications. The first is clinical biofeedback the second physiological signal-driven hands-free human–machine interaction. The same signal preprocessing and presentation requirements exist for both applications. It is for this reason J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 863 Fig. 1. A biofeedback-based interactive interface. that we examine technological developments in both application fields together grouped as before in terms of key physiological signals. 3.1.1. EMG-based applications Established as a clinical technique more than 20 years ago (Basmajian, 1977) EMG biofeedback is a popular method for training patients to regain muscle control lost due to accident or illness (Bowman, 1997). Volitional control of EMG is a promising means towards hands-free human–computer interaction (Knapp and Lusted, 1990; Rosenberg, 1998) and facial EMG has previously been used as a simple hands-free control signal for quadriplegic users (Lusted, 1996). Electronic prosthetic arms (Saridis and Gootie, 1982; Kelly 1990) are actually are biocybernetically adaptive systems that use neural networks to transform detectable EMG into control signals. However, biofeedback training is key to a successful outcome in terms of getting new amputees to accept and gain control of prosthetic limbs (Lake, 1997). EMG is the most well understood and immediately promising physiological signal source for hands-free human–machine interaction. This is due mostly to muscle control being a skill inherent in all able-bodied individuals. As we shall see later there are other signals used commonly in clinical biofeedback training that are also being explored as potential hands-free control signal sources. 3.1.2. EEG-based applications Beyond its role in physiological rehabilitation, biofeedback is used increasingly as part of the treatment for a growing number of psychophysiological disorders (i.e. disorders 864 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 with no clear physical cause). Conditions for which EEG- or neurofeedback is an applied technique include: † Attention Deficit Disorder (ADD) (Tansey, 1993; Lubar, 1995) † Addictions (Peniston and Kulkosky, 1989; Denney et al., 1991; Fahrion et al., 1992; Saxby and Penniston, 1995) † Anxiety (Hare et al., 1982) † Depression (Baehr et al., 1997) † Post-Traumatic Stress Disorder (PTSD) (Peniston and Kulkosky, 1991) † Sleep disorders (Bell, 1979). For each condition, a training protocol is applied which has been found through empirical investigation to contribute to the amelioration of that condition in other subjects. For example, research has identified signatory dysfunction in the EEG of sufferers of Attention Deficit Disorder (ADD) corresponding to increased activity in the theta band and reduced activity within the beta band (Lubar, 1997). A popular protocol for ADD therefore involves using PC-based biofeedback to train subjects to suppress theta activity and enhance beta activity. In recognition of the fact that the majority of sufferers of ADD are children, clinical biofeedback systems such as the Neurocybernetics II (http://www.eegspectrum.com—last accessed 15/06/04) enable feedback to be given in the form of interactive computer games. These are often goal-oriented with the subject being rewarded for achieving desirable signal characteristics. As ADD training protocols require the simultaneous training of multiple components of a subject’s EEG multichannel sensing devices are required. Commercial physiological sensing devices such as the Interactive BrainWave Visual Analyser (www.ibva.com—last accessed 15/06/04) can be used to detect multiple channels of the same physiological signal. Other physiological sensing systems like BioMuse (www.biocontol.com—last accessed 15/06/04), Neurocybernetics II and ProComp (www.thoughttechnology.com— last accessed 15/06/04) can be configured to detect a range of physiological data. All of these devices are serially connected computer peripherals designed to stream data in realtime to representative, responsive user interfaces. A separate body of research is emerging explicitly concerned with EEG-based handsfree human–machine interaction. Despite existing research being carried out mainly in the fields of rehabilitation and disabled user access, brain–computer interaction is an area of growing interest within the HCI community. As both biofeedback and biocybernetic adaptation are under investigation as means toward realisation of BCI this topic is dealt with separately in Section 3.2. 3.1.3. Applications based on multiple physiological signals As any single aspect of detectable human physiology may by itself prove unreliable as a hands-free control signal, multichannel and multisignal devices lend themselves to the possibility of training combinations of physiological signals in order to find a more reliable combi-signal. Applications which employ combinations of physiological signals include EEG/EMG combined hands-free human–computer interaction (Junker et al., J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 865 1995; Nelson et al., 1996) and wearable devices for biofeedback-based therapy (Paradiso et al., 2004; Wilhelm, 2002) and health monitoring (Gerasimov, 2003). The biofeedback systems described have all been conceived with the same aim—to assist an individual in the training for control of one or more physiological parameters. Regardless of whether the training itself is the reason for creating the system—as is the case with clinical biofeedback applications—or whether the training is a step toward some further goal such as hands-free interaction these physiologically interactive systems share the following features: (1) The need to retrieve physiological data from a dedicated electronic sensing peripheral (2) The requirement to pre-process physiological data in order to make it suitable for presentation (3) The need to present physiological data to a subject in a manner suitable for learning to take place Beyond the requirement for a suitable sensing device, the key defining feature of biofeedback-based physiological computing systems is the application-specific method of information display. The interface is crucial to the effectiveness of biofeedback training, i.e. how psychophysiological data is represented to the user and whether the biofeedback interface engages the user to facilitate training. We will revisit these important as part of the research agenda presented towards the end of this paper. 3.2. Biocybernetic adaptation Biocybernetic adaptation refers to the modification of system’s functionality or appearance based on the real-time measurement of psychophysiology. Rather than feeding psychophysiological data back to a subject, a biocybernetically adaptive systems use the changing psychophysiological state of the user in order to change its own functionality and/or appearance (Fig. 2). For example, a biocybernetic system may detect user fatigue based on long duration eyeblinks and increase font size or screen contrast in order to ease visual fatigue. Alternatively, the system may offer on-screen help based on the detection of user frustration, as measured by increased heart rate or blood pressure. Biocybernetic adaptation may be important for safety-critical performance in transport domains such as flight management and air traffic control. Prinzel (2002) identified a range of hazardous states of awareness, such as stress, high anxiety, boredom, absorption, fatigue and inattention, which may jeopardise performance quality and increase the risk of accident. The application of biocybernetic control to adaptive automation provides one example of this type of system. The introduction of system automation has the potential to enable greater control over complex systems and to compensate for inherent limitations on human performance (Woods, 1996). Despite these advantages, automation has been associated with several undesirable consequences including: deskilling, operator complacency, increased mental workload, boredom and new types of errors (Parasuraman and Riley, 1997; Woods, 1996). Adaptive automation offers a solution to these drawbacks. The adaptive approach permits a dynamic adjustment of system automation based on real-time analysis of operator performance or 866 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 Fig. 2. A biocybernetically adaptive user interface. the task scenario, i.e. switching between manual and automatic control is controlled by the system. It has been suggested that adaptive automation may be controlled by the analysis of psychophysiology in real time (Byrne and Parasuraman, 1996). Biocybernetic control via psychophysiology has a number of advantages over alternative methods of switching between manual and automatic control: (a) Psychophysiological activity is covert and does not require the operator to perform any additional tasks (b) Psychophysiological data is continuously available to the system whereas behavioural triggers may be discrete and intermittent (c) If a system was completely automated, psychophysiology could provide an assessment of operator state in the absence of overt behaviour. A prototype biocybernetic adaptive system was developed by NASA which simulated performance within an aerospace environment (Prinzel et al., 1995). This system recorded spontaneous EEG activity, which was expressed as a ratio measure (beta/(alphaCtheta)— the authors arguing that the proportion of lower frequency EEG signals to those in the beta range could be used to indicate the level of operator engagement in the task (Pope et al., 1995). This EEG signal was relayed to software that functioned as an adaptive controller during real-time performance on a laboratory task. If engagement declined, automated task components were switched to manual control, i.e. the switch to manual control should re-engage the operator. If the engagement index continued to increase, the adaptive controller assumed that the operator was stressed by the task and would proceed to automate the task. This negative control loop uses the adaptive control of automation to stabilise user engagement at a level that avoids both extremes of boredom and stress. J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 867 Further research on biocybernetic adaptation has produced several significant findings: † the use of negative feedback control produced a higher frequency of switching between automatic and manual modes compared to positive feedback control (Freeman et al., 1999) † superior performance and increased task engagement when using a biocybernetic system for sustained task performance periods (Freeman et al., 2000) † increased performance and reduced subjective mental workload when using biocybernetic adaptation (Prinzel et al., 2000). The NASA group recently extended their range of measures for biocybernetic adaptation to include event-related potentials and heart-rate variability (Prinzel et al., 2003). The reader is directed to Scerbo et al. (2003) for a recent review. The design logic behind biocybernetic adaptation will undoubtedly be exported to future generations of computer games. These games are currently designed to challenge, engage and stimulate the user, but the optimal level of challenge, etc. may vary considerably, both on an interand intra-person basis. Psychophysiological input provides a source of real-time adaptability for computer games, to dynamically adjust game difficulty to maximise user engagement, a hypothesis supported by early research in this area (Gilleade and Allanson, 2003). An application of increasing importance to the HCI community is affective computing (Picard, 1997). The goal of affective computing is to realise, within the machine, contextual understanding of emotional and empathetic human responses. Picard indicates that physiological monitoring alone is insufficient for detecting the valence of emotion. Despite this, frustration has been identified through off-line feature extraction from GSR and blood pulse volume (BVP) alone, using Hidden Markov Models (Fernandez, 1997). Recognition of the wider range of human emotions by machines will depend on assessment of a range of physiological markers as well as examination of factors such as facial expression, vocal intonation, posture and gesture. In all probability affective computers will also require environmental and task related data in order to make valid decisions about human emotional state (Picard and Klein, 2002). 3.3. Brain–computer interfacing Brain–Computer Interfaces (BCI) represent channels for direct command and communication between psychophysiology and an output device, e.g. control of a cursor (Wolpaw et al., 2002). In its simplest form, a BCI may involve direct correspondence between psychophysiological activity and computer output. The focus of current BCI research is to develop systems to help people with disabilities, e.g. spinal cord injuries, brain stem strokes. The aim of these systems is to translate volitional intention, which may be completely covert, into a computer-mediated process of selection and action. The development of a BCI is based on the reliable identification of recognisable and consistent psychophysiological changes with covert processes of perception and self-determination. The fidelity of this relationship between psychophysiology and intention/perception underlies the efficacy of the system. A BCI prototype has been developed based on event-related desynchronisation (ERD) in the EEG (Wolpaw et al., 2000). Users are required to control mu- or beta-rhythm amplitude 868 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 in order to move a computer cursor in one or two dimensions. Potential users must submit to a regime of biofeedback training in order to operate this BCI. Most employ motor imagery (e.g. they imagine hand movements) to control mu/beta amplitude and training may involve 2–3 sessions of 40-min duration over a period of 2–3 weeks to achieve significant control (Wolpaw et al., 2002). This approach has also been used to permit a tetraplegic patient to control an electrical hand orthosis fitting (Pfurtscheller et al., 2000, 2003). In order to be effective, a BCI must communicate information to the output device as accurately and efficiently as possible. If a BCI is deemed laborious and unrewarding, which may be a particular risk during the training phase, the user will be disinclined to use these systems to their full potential. Therefore, the rates of information transfer associated with BCI systems are an important determinant of user acceptance. For example, Farewell’s P300-based system claims a communication rate of five items per minute, whereas Wolpaw’s ERD-based system claims an information transfer rate of 20–25 bits/min when one-dimensional control was adopted (McFarland et al., 2000). The evaluation of information transfer rates is complicated by how many dimensions of control are available. This is particularly apparent if we consider the use of BCI to drive prosthetic devices where velocity and three dimensions of movement may be possible. One feature common to all BCIs is the task-specific nature of their interfaces. Where some clinical biofeedback applications incorporate highly abstract signal representation mechanisms, such as disembodied musical tones, BCIs require meaningful, goal-driven signal representation. A number of current BCI systems present soft keyboards as intermediate interfaces to speech synthesisers for physically disabled users. Beyond presentation of a keyboard, interface requirements can be quite diverse, with some exploratory systems (Keirn et al., 1990) requiring an interface that displays the results from a network classifier not to a subject, but to a professional observer. At first glance, systems such as the one developed by Pfurtscheller et al. (1997) focus on the classifying of brain signals, without much of a role for the systems’ interface. However, in systems requiring the training of neural network classifiers the interface plays a vital operational role in cueing the user to perform actions at intervals suitable to the processing capabilities of the classifier. Issues such as extended training times, the potentially limited applicability of direct brain to machine communication, and the unknown effects of adopting this method of human–machine interaction in real-world situations may discourage many from pursuing BCI. Indeed, when fully explored, brain–computer interface technology may be found to be applicable only in limited situations. However, the importance of continuing exploration of BCI for users with limited physical capabilities is immeasurable. 4. A research agenda for physiological computing The emergence of new computing paradigms demands the development of new tools to support the evolution of systems adhering to these models. In order to realise development support for physiological computers we need to: (a) incorporate basic psychophysiological theory into the design of these systems J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 869 (b) identify the signal processing requirements for physiological sensing technologies and (c) design and implement device-independent software tools and architectures suitable for creating physiological computers. We will now consider each of these requirements in turn. 4.1. Measurement issues Psychophysiology is based on the premise that psychological processes and states are accompanied by changes in physiological activity. Therefore, it is important that psychological processes and physiological activity are tightly coupled. The identification of facial expressions via EMG patterns from the facial muscles is an example of close coupling between psychology and physiology (Cacioppo et al., 1990). This coupling is more difficult to maintain when physiology is used to measure psychological states. The success of biocybernetic adaptation depends on the identification of appropriate variables to evoke desirable and undesirable psychological states. In the case of hazardous states of awareness, this means basic research into the psychophysiology of cognitive-energetical variables such as: mental workload (Wilson, 2002), lapses of alertness (Mekeig and Inlow, 1993), anxiety (Mueller, 1992) and boredom (Davies et al., 1983). The application of biocybernetic adaptation to computer games requires the characterisation of relevant psychological states such as threat and challenge (Blascovich et al., 1999; Carroll et al., 1986a,b; Mathias and Stanford, 2003; Quigley et al., 2002). Similarly, the development of affective computing requires an understanding of how psychophysiology may be used to represent affective dimensions (Christie and Friedman, 2004). The selection of psychophysiological variables that adequately operationalise covert, psychological processes is fundamental for the success of physiological computing systems. Researchers must assess the viability of individual psychophysiological variables or groups of variables according to criteria of sensitivity and diagnosticity (O’Donnell and Eggemeier, 1986). A sensitive psychophysiological variable must be capable of discriminating significant variations in the psychological process under consideration. For example, a sensitive variable would distinguish between high, medium and low levels of mental workload, or discriminate between low boredom and high boredom, or classify extreme anxiety from moderate levels of anxiety. The sensitivity of any psychophysiological variable must be assessed with reference to the fidelity of biocybernetic control required by the system. If a designer is looking for a simple binary input, then an EMG spike may suffice; however, if the system is constructed to adapt to five quantitatively distinct levels of anxiety, then it is important to find a psychophysiological operationalisation to adequately represent this continuum. Therefore, the sensitivity required from psychophysiological input is determined by the fidelity of control demanded by the physiological computing system. The diagnosticity of a psychophysiological variable represents the precision of his operationalisation. Psychophysiological data such as heart rate represent both the influence of psychological processes (such as anxiety) and the homeostatic control of the internal physiological environment (e.g. control of body temperature). The criterion of 870 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 diagnosticity refers to the extent to which a psychophysiological variable target a specific psychological process as opposed to other related processes. For example, if the system is designed to detect and respond to levels of anxiety, it must be sufficiently diagnostic to discriminate the state of anxiety from other mood states such as high excitement or happiness. Similarly, if a BCI device is designed to produce right-hand and left-hand inputs, the system must be able to distinguish between mu-rhythms associated with lateral and contra-lateral patterns of motor activity. The level of diagnosticity determines the frequency of inappropriate or unintended responses from the physiological computing system. Psychophysiological measures may be ‘contaminated’ by behavioural activity (e.g. movement, physical exertion, speaking) and environmental factors (e.g. temperature, humidity, noise). These factors may distort physiological activity, effectively destroying the association between psychology and physiology. This signal/noise ratio represents the reliability of the psychophysiological measure or susceptibility to confounding influences. Measures that are inherently unreliable are a cause for concern because they require additional processing of the raw digital signal to assess reliability of data (see Section 4.2). In addition prolonged periods of confounded data may compromise the sensitivity of any given measure. Psychophysiological measurement often means that humans are attached to computer hardware via various sensor apparatus, e.g. electrodes. There are problems of intrusiveness if the application of sensor apparatus causes discomfort to the user or restricts freedom of movement. This may be a particular problem for EEG data capture or psychophysiological measures such as GSR where electrode site placement is restricted to the palms and the soles of the feet. It is anticipated that problems of intrusiveness will be resolved via the introduction of new sensors (see Section 4.3) and the advancement of ambulatory apparatus for data capture (Fahrenberg, 1996). These criteria should be used to guide the selection of measures to be used as input to a physiological computing system. A reliable and usable system should aim to maximise the sensitivity, diagnosticity and reliability of candidate measures whilst keeping the intrusiveness to a minimum. 4.2. Signal processing issues The purpose of a physiological computing system is to transform a raw, analogue signal from the human body into a variable component within a computerised command protocol. This transformation involves a number of discrete stages from extraction of the raw signal to analysis and integration of the data as a computer input. The analogue signal must initially be transformed into a digital signal during the process of extraction. This is the purpose of commercially available hardware for psychophysiological data capture. The most important characteristic of extraction is setting the correct signal sample rate. For example, if one is measuring changes in eye blink duration then one must sample the EOG at 1000 Hz to achieve sufficient temporal resolution. However, if a system required respiration rate expressed as breaths per minute, a lower sample rate of 2–5 Hz would suffice. The selection of the correct sample rate depends on the characteristics of both the raw signal and the translated signal. J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 871 Once the digital signal has been captured, the integrity of the signal must be assessed and transformed into the desired unit of measurement. This process of translation involves an on-line analysis of the raw digital signal. For example, a simple threshold detection algorithm may be used to detect the R-peak of an ECG signal and the time interval between successive R-peaks is recorded as the inter-beat interval, which is a measure of heart rate. Other variables may rely on spatial filters in order to filter selected bandwidths of interest, e.g. if wave magnitudeZx or wave frequencyZy, filter as alpha waves from the EEG. The process of spatial filtering may involve Fast Fourier Transforms (FFT) which expresses activity within selected bandwidths as a power value. The FFT procedure has the disadvantage of reducing the temporal resolution of the signal but may be applied across a range of psychophysiological measures. The purpose of all these techniques is to express the raw digital signal from the human body in conventional units of measurement, e.g. heart rate, respiration rate, alpha power, eye blink duration. The detection of artefacts in the data may occur either before or after this stage. For example, it is known that eye blinks and eye movements may confound EEG data collected from frontal sites on the scalp; therefore, the influence of eye movements on EEG activity may be filtered prior to spatial filtering or FFT analysis (Croft and Barry, 2000). Alternatively, data may be checked for aberrant patterns and confounds following the translation from raw digital signal to the transformed signal, e.g. reject inter-beat intervals above 2000 ms. The purpose of the translation process is to render data reliable and meaningful prior to further analyses. Translated data expressed in psychophysiological units (e.g. heart rate, breaths per minute, mean GSR, alpha power) must be analysed to assess its behavioural significance. It is assumed that a physiological computing system is constructed with ‘built-in expertise’ in the form of feature extraction protocols, mathematical rules and/or neural networks. In the first instance, the system must recognise any meaningful deviations in the translated data, which may represent changes in psychological state or intention to move a limb, etc. For example, a 40% increase of heart rate would be indicative of high levels of stress or anxiety. This type of detection demands the integration of baseline data within the computing system that is tailored to the individual user or representative of population norms. This analysis requires the development of algorithms to recognise specific psychological states and to quantify variations in those states. These algorithms encompass both the direction and magnitude of expected psychophysiological change. Algorithm definition and construction must be based on research into the classification of different psychophysiological states. For example, stepwise discriminant analysis was used to distinguish between task types (e.g. visual, verbal, memory, mental arithmetic, etc.) and task difficulty (Wilson and Fisher, 1991, 1996). Recent research demonstrated that the superiority of neural network approaches to distinguish between different levels of task difficulty (Gevins et al., 1998; Laine et al., 2002; Wilson and Russell, 2003). This capability to classify and diagnose incoming data must be formalised for the computer system to recognise anger, the movement of a limb or the shift from an easy to a difficult task. The integration of psychophysiological data into a command protocol demands that the system makes an appropriate response based on the detection of psychophysiological 872 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 changes. For example, one biocybernetic prototype activated system automation in response to stress and deactivated automation if the user exhibited low levels of task engagement (Prinzel et al., 2000). A command protocol will determine if the computer makes a response and if so, the direction and magnitude of that response. A large physiological change in the area of a 40% increase in heart rate would probably demand a response, but the decision to respond involves a degree of fuzzy logic when physiological changes are modest. This conversion from computerised pattern recognition into a response from the system that matches user expectations represents a particular challenge for human factors. 4.3. Sensors, architectures, tools Physiological computers are at an important stage in their evolution. The minimum configuration for a physiological computer is a PC and off-the-shelf physiological sensors. However, the current requirement for ad hoc development is hampering progress in physiological computing research. What is required is the kind of development support that has enabled the WIMP paradigm to proliferate. The concerns to be addressed in solution to this problem are twofold; firstly, the need to abstract over multiple physiological sensing devices in order to support the notion of device-independence; secondly, the need to handle the continuous sensing capabilities of physiological sensors. The first issue is traditionally addressed through identification of high level models of interaction, the second by specialised tools and architectures. Key to the evolution of any class of interactive computer system is identification of models that facilitate abstracted reasoning about interaction. Sensor-based interaction does not rely on the recognition of deliberate, intentional user-generated gestures. Biofeedback-based interfaces operate due to a real-time feedback loop which enables the user to make associations between attempts to influence interface components by conscious control and the behaviour of those interface components. With adaptive physiological computing applications, however, aspects of the interface change in response to unintentional (in fact, largely unconscious) generation of the same data, as it relates in this instance to changing physiological status as a by-product of some other activity. How do we define interaction when it can mean anything from sitting in front of a screen training some component of your EEG to interacting with a computer which can tell how you are feeling? As the dominant input sensors are so unlike the mechanical navigation and selector tools we are used to look at the Structural Coupling Paradigm (McMillan et al., 1995). Fig. 3 shows a version of the Structural Coupling Paradigm modified to describe the interactions between human and physiological computer. The mechanical linkage is broken and the issue of interpreting user data is dealt with by means of an intelligent sensor. This sensor incorporates data interpretation mechanisms. So, for example, in a biofeedback-based system where the user is exploring strategies to raise her heart rate, the intelligent sensor’s interpretation component will track heart beats with a view to notifying (command) the interface when the rate falls below a pre-defined threshold value. Conversely, in a system where the user’s spontaneous EEG is being used to affect changes J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 873 Fig. 3. The structured coupling paradigm (after McMillan et al., 1995). in the interface, the sensor’s interpretation component will collect data from a range of electrode placements across the scalp and use a neural classifier in order to extract features (commands) from the data stream. Interaction techniques provided in support of existing paradigms provide an abstraction over hardware. This is desirable in physiological computing applications also, as we do not want our application to be coupled too closely to any particular manufacturer’s sensor. Of course the nature of interaction techniques for physiological computing will be diverse— from real-time polygraphic representations of physiological data to interactive 3D models of prosthetic arms. The full range of possibilities will emerge from careful review of the full range of interactive applications to which physiological information can usefully be applied as an input source. Where interface widgets have traditionally been designed to react to a signal discrete signal, physiological computer interface components may need to deal with various types of data (each, as we have seen, with its own particular characteristics). For this reason, it might be necessary to provide a layer of signal pre-processing between the intelligent sensors and the interface in order to limit the complexity of interaction techniques. The implications of this are discussed further in Allanson (2000). 5. Conclusions This paper has provided a primer on psychophysiology for the general reader and proposed that there are two categories of system to which all physiological computing solutions adhere—biofeedback-based systems and systems that are biocybernetically adaptive. These systems cover a range of applications from health monitoring and safety 874 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 applications through to hands-free control. From this starting point a co-ordinated solution to the future design and implementation of physiological computers can be envisaged. The development of physiological computing systems will be a multidisciplinary programme of research. The fidelity of the computerised response depends on the precision of psychophysiological operationalisation and the availability of sophisticated protocols for measurement and analysis. The designers of physiological systems must select their measures appropriately in order to maximise sensitivity, diagnosticity and reliability of data input. The acceptability of this technology rests on two primary factors, the intrusiveness of sensor apparatus and the level of trust engendered by system usage. Interaction with physiological computers should be intuitive and therefore, the system response should match the expectations of the user. The perceived accuracy of this conversion from covert physiological change to overt system response is an important element of system evaluation. This paper shows that our technological and physiological knowledge has reached a point where it is possible to realise simple integrated physiological computing solutions. The next step is to co-ordinate efforts in order to provide development support for physiological computing. As the technological requirements already available us through assessment of existing systems we can begin to envisage toolkits architectures and protocols for physiological computing. Development of intelligent physiological sensors is a longer-term goal and must be informed by further work within the field of psychophysiology. A mechanism must be found for collating relevant research and commissioning new studies in the fields of psychophysiology, rehabilitative medicine (biofeedback practitioners, clinical psychologists, prosthetists, BCI researchers), human factors and ergonomics, artificial intelligence and HCI. References Aasman, J., Mulder, G., Mulder, L.J.M., 1987. Operator effort and the measurement of heart rate variability. Human Factors 29 (2), 161–170. Allanson, J., 2000. Developing support of electrophysiologically-interactive computer systems. PhD Thesis, Lancaster University, UK. Andreassi, J.L., 1995. Psychophysiology: Human Behaviour and Physiological Response. Erlbaum, Hillsdale, NJ. Backs, R.W., Selijos, K.A., 1994. Metabolic and cardiorespiratory measures of mental effort: the effects of level of difficulty in a working memory task. International Journal of Psychophysiology 16, 57–68. Baehr, E., Rosenfeld, J.P., Baehr, R., 1997. The clinical use of an alpha asymmetry protocol in the neurofeedback treatment of depression: two case studies. Journal of Neurotherapy 2 (3), 10–23. Basmajian, J.V., 1977. In: Schwartz, G.E., Beatty, J. (Eds.), Learned Control of Single Motor Units. Biofeedback Theory and Research. Academic Press, New York. Beatty, J., 1982. Task-evoked pupillary responses, processing load and the structure of processing resources. Psychological Bulletin 91 (2), 276–292. Bell, J.S., 1979. The use of EEG theta biofeedback in the treatment of a patient with sleep-onset insomnia. Biofeedback and Self-Regulation 4 (3), 229–236. Berntson, G.G., Bigger, J.T., Eckberg, D.L., Grossman, P., Kaufman, P.G., Malik, M., et al., 1997. Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology 34, 623–648. Blascovich, J., Berry-Mendes, W., Hunter, S.B., Salomon, K., 1999. Social ‘facilitation’ as challenge and threat. Journal of Personality and Social Psychology 77 (1). J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 875 Boiten, F., 1998. The effects of emotional behaviour on components of respiratory cycle. Biological Psychology 49, 29–51. Boutcher, S.H., Nugent, F.W., McLaren, P.F., Weltman, A.L., 1998. Heart period variability of trained and untrained men at rest and during mental challenge. Psychophysiology 35, 16–22. Bowman, T., 1997. VR meets physical therapy. Communications of the ACM 40 (8), 59–60. Byrne, E., Parasuraman, R., 1996. Psychophysiology and adaptive automation. Biological Psychology (42), 1996, 249–268. Cacioppo, J.T., Bush, L.K., Tassinary, L.G., 1990. Microexpressive facial actions as a function of affective stimuli: replication and extension. Personality and Social Psychology Bulletin 18, 515–526. Cacioppo, J.T., Tassinary, L.G., Berntson, G.G., 2000. Handbook of Psychophysiology. Cambridge University Press, Cambridge. Calvert, S.L., Tan, S., 1994. Impact of virtual reality on young adult’s physiological arousal and aggressive thoughts: interaction versus observation. Journal of Applied Developmental Psychology 15, 125–139. Carroll, D., Turner, J.R., Hellawell, J.C., 1986a. Heart rate and oxygen consumption during active psychological challenge: the effects of levels of difficulty. Psychophysiology 23, 174–181. Carroll, D., Turner, J.R., Prasad, R., 1986b. The effects of level of difficulty of mental arithmetic challenge on heart rate and oxygen consumption. International Journal of Psychophysiology 4, 167–173. Christie, I.C., Friedman, B.H., 2004. Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. International Journal of Psychophysiology 51, 143–153. Croft, R.J., Barry, R.J., 2000. Removal of ocular artifact from the EEG: a review. Neurophysiology Clinical 30, 5–19. Davies, C., Harrison, J., 1996. Osmose: towards broadening the aesthetics of virtual reality. Computer Graphics 1996. Davies, D.R., Shackleton, V.J., Parasuraman, R., 1983. Monotony and boredom. In: Hockey, G.R.J. (Ed.), Stress and Fatigue in Human Performance. Wiley, Chichester, pp. 1–25. Denney, M.R., Baugh, J.L., Hardt, H.D., 1991. Sobriety outcome after alcoholism treatment with biofeedback participation: a pilot inpatient study. International Journal of Addiction 26 (3), 335–341. Fahrenberg, Myrtek, 1996. Ambulatory Assessment: Computer-assisted Psychological and Psychophysiological Methods in Monitoring and Field Studies. Hogrefe and Huber, Gottingen, pp. 3–19. Fahrion, S.L., Walters, E.D., Coyne, L., Allen, T., 1992. Alterations in EEG amplitude, personality factors, and brain electrical mapping after alpha–theta brainwave training: a controlled case study of an alcoholic in recovery. Alcoholism, Clinical and Experimental Research 16 (3), 547–552. Farewell, L.A., Donchin, E., 1988. Talking off the top off your head: toward a mental prosthesis utilizing eventrelated brain potentials. Electroencephalography and Clinical Neurophysiology 70, 520–523. Fernandez, R., 1997. Stochastic modelling of physiological signals with hidden Markov models: a step toward frustration detection in human–computer interfaces. Masters Thesis, Media Laboratory, MIT. Freeman, F.G., Mikulka, P.J., Prinzel, L.J., Scerbo, M.W., 1999. Evaluation of an adaptive automation system using three EEG indices with a visual tracking task. Biological Psychology 50, 61–76. Freeman, F.G., Mikulka, P.J., Scerbo, M.W., Prinzel, L.J., Clouatre, K., 2000. Evaluation of a psychophysiologically controlled adaptive automation system, using performance on a tracking system. Applied Psychophysiology and Biofeedback 25 (2), 103–115. Gerasimov, V., 2003. Every sign of life. PhD Thesis, MIT. Gevins, A., Smith, M.E., 2003. Neurophysiological measures of cognitive workload during human–computer interaction. Theoretical Issues in Ergonomic Science 4 (1/2), 113–121. Gevins, A., Smith, M.E., Leong, H., McEvoy, L., Whitfield, S., Du, R., et al., 1998. Monitoring working memory load during computer-based tasks with EEG pattern recognition models. Human Factors 40 (1), 79–91. Gilleade, K., Allanson, J., 2003. A Toolkit for exploring affective interface adaptation in videogames, Proceedings of the Human–Computer Interaction International (HCII03), Crete, Greece, 2003. Hare, J.F., Timmons, B.H., Roberts, J.R., Burman, A.S., 1982. EEG alpha-biofeedback training: an experimental technique for the management of anxiety. Journal of Medical Engineering and Technology 6 (1), 19–24. Junker, A., Berg, C., Schnider, P., 1995. Evaluation of the cyberlink interface as an alternative human operator controller. Technical Report No. AC/CF-TR-1995-0011, Wright-Patterson Air Force Base, OH. 876 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 Kelly, M.F., Parker, P.A., Scott, R.N., 1990. The application of neural networks to myoelectric signal analysis: a preliminary study. IEEE Engineering in Medicine and Biology 37, 221–229. Knapp, R.B., Lusted, H.S., 1990. Bioelectric controller for computer music applications. Computer Music Journal 14 (1), 42–47. Kramer, A.F., 1987. Event-related brain potentials. In: Gale, A., Christie, B. (Eds.), Psychophysiology and the Electronic Workplace. Wiley, London, pp. 197–222. Laine, T.I., Bauer, K.W., Lanning, J.W., Russell, C.A., Wilson, G.F., 2002. Selection of input features across subjects for classifying crewmember workload using artificial neural networks. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans 32 (6), 691–704. Lake, C., 1997. Effects of prosthetic training on upper-extremity prosthesis use. Journal of Prosthetics and Othotics 9 (1), 3–9. Light, K., 1981. Cardiovascular responses to effortful active coping: implications for the role of stress in hypertension development. Psychophysiology 18 (3), 216–225. Lubar, J.F., 1995. In: Schwartz, M.S. (Ed.), Neurofeedback for the Management of Attention Deficit/ Hyperactivity DisordersBiofeedback: A Practitioner’s Guide. Guilford Press, New York, pp. 493–525. Lubar, J.F., 1997. Neocortical dynamics: implications for understanding the role of neurofeedback and related techniques for the enhancement of attention. Applied Psychophysiology and Biofeedback 22 (2), 111–126. Lusted, H.S., Knapp, R.B., Benjamin, R., 1996. Controlling Computers with Neural Signals. Scientific American, pp. 58–63. Malmo, R.B., Malmo, H.P., 2000. On electromyographic (EMG) gradients and movement-related brain activity: significance for motor control, cognitive functions, and certain psychopathologies. International Journal of Psychophysiology 38, 143–207. Martin, I., Venables, P.H., 1980. Techniques in Psychophysiology. Wiley, New York. Mathias, C.W., Stanford, M.S., 2003. Impulsiveness and arousal: heart rate under conditions of rest and challenge in healthy males. Personality and Individual Differences 35, 355–371. McFarland, D.J., Sarnacki, W.A., Vaughan, T.M., Wolpaw, J.R., 2000. EEG-based interface communication effect of target number and trial length on information transfer rate. Society of Neuroscience Abstracts 26, 1228. McMillan, G.R., Eggleston, R.G., Anderson, T.R., 1995. Nonconventional controls. In: Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics. Wiley, New York, pp. 729–771. Mekeig, S., Inlow, M., 1993. Lapses in alertness: coherence of fluctuations in performance and EEG spectrum. Electroencephalography and Clinical Neurophysiology 86, 23–25. Mueller, J.H., 1992. Anxiety and performance. In: Smith, A.P., Jones, D.M. (Eds.), Handbook of Human Performance, vol. 3. Academic Press, London, pp. 127–160. Mulder, G., 1986. The concept and measurement of mental effort. In: Hockey, G.R.J., Gaillard, A.W.K., Coles, M.G.H. (Eds.), Energetical Issues in Research on Human Information Processing. Martinus Nijhoff, Dordrecht, pp. 175–198. Nelson, T.W., Hettinger, L.J., Cunningham, J.A., Roe, M.M., Lu, L.G., Haas, M.W., Dennis, L.B., Pick, H.L., Junker, A., 1996. Brain–body actuated control: assessment of an alternative control technology for virtual environments, 1996 Image Conference, Scottsdale, AZ 1996, pp. 225–232. O’Donnell, R.D., Eggemeier, F.T., 1986. Workload assessment methodology, In: Boff, K., Kaufman, L., Thomas, J. (Eds.), Handbook of Human Perception and Performance, vol. 2. Wiley, New York, pp. 42.41–42.49. Olson, P.R., 1995. Definitions of Biofeedback and Applied Psychophysiology Biofeedback: A Practitioner’s Guide. In: Schwartz, M.S. (Ed.),. The Guilford Press, New York, pp. 27–31. Paradiso, J., Morris, S., Benbasat, A., Asmussen, E., 2004. Interactive therapy with instrumented footwear. Electronic Proceedings CHI 2004, pp. 1341 (CD2). Parasuraman, R., Riley, V., 1997. Humans and automation: use, misuse, disuse, abuse. Human Factors 39, 230–253. Partala, T., Surakka, V., 2003. Pupil size variation as an indication of affective processing. International Journal of Human–Computer Studies 59, 185–198. Peniston, E.G., Kulkosky, P.J., 1989. Alpha–theta brainwave training and beta-endorphin levels in alcoholics. Alcoholism, Clinical and Experimental Research 13 (2), 271–279. J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 877 Peniston, E.G., Kulkosky, P.J., 1991. Alpha–theta brainwave neuro-feedback therapy for Vietnam veterans with combat-related post-traumatic stress disorder. Medical Psychotherapy 4, 47–60. Pfurtscheller, G., Neuper, C., Edlinger, A.G., 1997. Foot and hand area mu rhythms. International Journal of Psychophysiology 26, 121–135. Pfurtscheller, G., Guger, C., Muller, G., Krausz, G., Neurper, C., 2000. Brain oscillations control hand orthosis in a tetraplegic. Neuroscience Letters 292, 211–214. Pfurtscheller, G., Muller, G., Pfurtscheller, J., Gerner, H.J., Rupp, R., 2003. ‘Thought’—control of functional electrical stimulation to restore hand grasp in a patient with tetraplegia. Neuroscience Letters 351, 33–36. Picard, R.W., 1997. Affective Computing. The MIT Press. Picard, W., Klein, J., 2002. Computers that recognise and respond to user emotion: theoretical and practical implications. Interacting with Computers 14, 141–169. Pope, A.T., Bogart, E.H., Bartolome, D.S., 1995. Biocybernetic system evaluates indices of operator engagement in automated task. Biological Psychology 40, 187–195. Porges, S.W., Byrne, E.A., 1992. Research methods for measurement of heart rate and respiration. Biological Psychology 34, 93–130. Prinzel, L.J., 2002. Research in hazardous states of awareness and physiological factors in aerospace operations (No. NASA/TM-2002-211444): NASA. Prinzel, L.J., Scerbo, M.W., Freeman, F.G., Mikulka, P.J., 1995. A bio-cybernetic system for adaptive automation, Human Factors and Ergonomics Society 39th Annual Meeting 1995. Prinzel, L.J., Freeman, F.G., Scerbo, M.W., Mikulka, P.J., Pope, A.T., 2000. A closed-loop system for examining psychophysiological measures for adaptive task allocation. International Journal of Aviation Psychology 10 (4), 393–410. Prinzel, L.J., Parasuraman, R., Freeman, F.G., Scerbo, M.W., Mikulka, P.J., Pope, A.T., 2003. Three experiments examining the use of electroencephalogram, event-related potentials, and heart-rate variability for real-time human-centred adaptive automation design (No. NASA/TP-2003-212442): NASA. Quigley, K.S., Barrett, L.F., Weinstein, S., 2002. Cardiovascular patterns associated with threat and challenge appraisals: a within-subjects analysis. Psychophysiology 39, 292–302. Rosenberg, R., 1998. Computing without mice and keyboards: text and graphic input devices for mobile computing. PhD Thesis, UCL, London. Saridis, G.N., Gootie, T.P., 1982. EMG pattern analysis and classification for a prosthetic arm. IEEE Engineering in Medicine and Biology 29, 403–411. Saxby, E., Penniston, E.G., 1995. Alpha–theta brainwave neurofeedback training: an effective treatment for male and female alcoholics with depressive symptoms. Journal of Clinical Psychology 51 (5), 685–693. Scerbo, M.W., Freeman, F.G., Mikulka, P.J., 2003. A brain-based system for adaptive automation. Theoretical Issues in Ergonomic Science 4 (1/2), 200–219. Siddle, D.A.T., Turpin, G., 1980. Measurement, quantification and analysis of cardiac activity. In: Martin, I., Venables, P.H. (Eds.), Techniques in Psychophysiology. Wiley, New York, pp. 139–246. Stern, J.A., Boyer, D., Schroeder, D., 1994. Blink rate: a possible measure of fatigue. Human Factors 36 (2), 285–297. Tansey, M.A., 1993. Ten-year stability of EEG biofeedback results for a hyperactive boy who failed fourth grade perceptually impaired class. Biofeedback and Self-regulation 18 (1), 33–44. Tattersall, A.J., Hockey, G.R.J., 1995. Level of operator control and changes in heart rate variability during simulated flight maintenance. Human Factors 37 (4), 682–698. Veltman, J.A., Gaillard, A.W.K., 1998. Physiological workload reactions to increasing levels of task difficulty. Ergonomics 41 (5), 656–669. Vertegaal, R., 2003. Attentive user interfaces. Communications of the ACM 2003. Verwey, W.B., Veltman, H.A., 1996. Detecting short periods of elevated workload: a comparison of nine workload assessment techniques. Journal of Experimental Psychology: Applied 2 (3), 270–285. Wiederhold, B.K., Jang, D.P., Kaneda, M., Cabral, I., Lurie, Y., May, T., Wiederhold, M.D., Kim, S.I., 2003. An investigation into physiological responses in virtual environments: an objective measurement of presence. In: Riva, G., Galimberti, C. (Eds.), Towards Cyberpsychology: Minds, Cognitions and Society in the Internet Age. IOS Press, Amsterdam. 878 J. Allanson, S.H. Fairclough / Interacting with Computers 16 (2004) 857–878 Wiener, N., 1961. Cybernetics: or Control and Communication in the Animal and the Machine. MIT Press, Cambridge, MA. Wientjes, C.J.E., 1992. Respiration in psychophysiology: methods and applications. Biological Psychology 34, 179–204. Wilhelm, F.H., 2002. Continuous electronic data capture of cardiopulmonary physiology, motor behavior and subjective experience with the LifeShirt: towards a comprehensive monitoring of affective states in real life. Workshop on Physiological Computing at CHI2002 2002. Wilson, G.F., 2002. An analysis of mental workload in pilots during flight using multiple psychophysiological measures. International Journal of Aviation Psychology 12 (1), 3–18. Wilson, G.F., Fisher, F., 1991. The use of cardiac and eye blink measures to determine flight segment in F4 crews. Aviation, Space and Environmental Medicine 62, 959–961. Wilson, G.F., Fisher, F., 1996. Cognitive task classification based upon topographic EEG data. Biological Psychology 40, 239–250. Wilson, G.F., Russell, C.A., 2003. Operator functional state classification using multiple psychophysiological features in an air traffic control task. Human Factors 45 (3), 381–389. Wilson, G.F., Swain, C.R., Ullsperger, P., 1999. EEG power changes during a multiple level memory retention task. International Journal of Psychophysiology 32, 107–118. Wolpaw, J.R., McFarland, D.J., Vaughan, T.M., 2000. Brain–computer interface research at the Wadsworth Center. IEEE Transactions on Rehabilitation Engineering 8, 222–225. Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M., 2002. Brain–computer interfaces for communication and control. Clinical Neurophysiology 113, 767–791. Woods, D.D., 1996. Decomposing automation: apparent simplicity, real complexity. In: Parasuraman, R., Mouloua, M. (Eds.), Automation and Human Performance: Theory and Application. Erlbaum, Hillsdale, NJ, pp. 3–17. Yamada, F., 1998. Frontal midline theta rhythm and eyeblinking activity during a VDT task and a video game: useful tools for psychophysiology in ergonomics. Ergonomics 41 (5), 678–688.