skip to main content
10.1145/3173574.3173856acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

The Index of Pupillary Activity: Measuring Cognitive Load vis-à-vis Task Difficulty with Pupil Oscillation

Published: 21 April 2018 Publication History

Abstract

A novel eye-tracked measure of the frequency of pupil diameter oscillation is proposed for capturing what is thought to be an indicator of cognitive load. The proposed metric, termed the Index of Pupillary Activity, is shown to discriminate task difficulty vis-a-vis cognitive load (if the implied causality can be assumed) in an experiment where participants performed easy and difficult mental arithmetic tasks while fixating a central target (a requirement for replication of prior work). The paper's contribution is twofold: full documentation is provided for the calculation of the proposed measurement which can be considered as an alternative to the existing proprietary Index of Cognitive Activity (ICA). Thus, it is possible for researchers to replicate the experiment and build their own software which implements this measurement. Second, several aspects of the ICA are approached in a more data-sensitive way with the goal of improving the measurement's performance.

References

[1]
Sylvia Ahern and Jackson Beatty. 1979. Pupillary Responses During Information Processing Vary with Scholastic Aptitude Test Scores. Science 205, 4412 (1979), 1289--1292.
[2]
Miyuki Azuma, Takehiro Minamoto, Ken Yaoi, Mariko Osaka, and Naoyuki Osaka. 2014. Effect of memory load in eye movement control: A study using the reading span test. Journal of Eye Movement Research 7, 5 (2014), 1--9.
[3]
Brian P. Bailey and Shamsi T. Iqbal. 2008. Understanding Changes in Mental Workload During Execution of Goal-directed Tasks and Its Application for Interruption Management. ACM Trans. Comput.-Hum. Interact. 14, 4, Article 21 (Jan. 2008), 28 pages.
[4]
Mike Bartels and Sandra P. Marshall. 2012. Measuring Cognitive Workload Across Different Eye Tracking Hardware Platforms. In ETRA '12: Proceedings of the 2008 Symposium on Eye Tracking Research & Applications. ACM, Santa Barbara, CA.
[5]
Jackson Beatty. 1982. Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources. Psychological Bulletin 91, 2 (1982), 276--292.
[6]
Jackson Beatty and Brennis Lucero-Wagoner. 2000. The Pupillary System. In Handbook of Psychophysiology (2nd ed.), John T. Cacioppo, Louis G. Tassinary, and Gary G. Bernston (Eds.). Cambridge University Press, 142--162.
[7]
Deborah A. Boehm-Davis, Wayne D. Gray, Leonard Adelman, Sandra Marshall, and Robert Pozos. 2003. Understanding and Measuring Cognitive Workload: A Coordinated Multidisciplinary Approach. Technical Report AFRL-SR-AR-TR-03-0407 (ADA417743); Grant #49620--97--1-0353. AFOSR, Arlington, VA.
[8]
Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1 (March 1994), 49--59.
[9]
Siyuan Chen and Julien Epps. 2014a. Efficient and Robust Pupil Size and Blink Estimation from Near-Field Video Sequences for Human-Machine Interaction. IEEE Transactions on Cybernetics 44, 12 (2014), 2356--2367.
[10]
Siyuan Chen and Julien Epps. 2014b. Using Task-Induced Pupil Diameter and Blink Rate to Infer Cognitive Load. Human-Computer Interaction 29, 4 (2014), 390--413.
[11]
Benjamin Cowley, Marco Filetti, Kristian Lukander, Jari Torniainen, Andreas Henelius, Lauri Ahonen, Oswald Barral, Ilkka Kosunen, Teppo Valtonen, Minna Huotilainen, Niklas Ravaja, and Giulio Jacucci. 2016. The Psychophysiology Primer: A Guide to Methods and a Broad Review with a Focus on Human-Computer Interaction. Foundations and Trends in Human-Computer Interaction 9, 3--4 (2016), 151--308.
[12]
Nicolas Debue and Cécile van de Leemput. 2014. What does germane load mean? An empirical contribution to the cognitive load theory. Frontiers in Psychology 5 (2014), 1099--1--1099--12.
[13]
Andrew T. Duchowski. 1997. Gaze-Contingent Visual Communication. Ph.D. Dissertation. Texas A&M University, College Station, TX.
[14]
Ralf Engbert and Reinhold Kliegl. 2003. Microsaccades uncover the orientation of covert attention. Vision Research 43 (2003), 1035--1045.
[15]
Randall W. Engle. 2002. Working Memory Capacity as Executive Attention. Current Directions in Psychological Science 11, 1 (2002), 19--23.
[16]
Randall W. Engle and Michael J. Kane. 2003. Executive Attention, Working Memory Capacity, and a Two-Factor Theory of Cognitive Control. Psychology of Learning and Motivation 44 (2003), 145--199.
[17]
Paul M. Fitts, Richard E. Jones, and John L. Milton. 1950. Eye Movements of Aircraft Pilots During Instrument-Landing Approaches. Aeronautical Engineering Review 9, 2 (1950), 24--29.
[18]
Eric Granholm, Robert F. Asarnow, Andrew J. Sarkin, and Karen L. Dykes. 1996. Pupillary responses index cognitive resource limitations. Psychophysiology 33, 4 (1996), 457--461.
[19]
Amara Graps. 1995. An Introduction to Wavelets. IEEE Computational Science & Engineering (1995), 50--61.
[20]
Eija Haapalainen, SeungJun Kim, Jodi F. Forlizzi, and Anind K. Dey. 2010. Psycho-physiological Measures for Assessing Cognitive Load. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (UbiComp '10). ACM, New York, NY, 301--310.
[21]
Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 50. 904--908.
[22]
Taylor R. Hayes and Alexander A. Petrov. 2016. Mapping and correcting the influence of gaze position on pupil size measurements. Behavior Reearch 48 (2016), 510--527.
[23]
Eckhard H. Hess and James M. Polt. 1964. Pupil Size in Relation to Mental Activity during Simple Problem-Solving. Science 143, 3611 (March 1964), 1190--1192.
[24]
Nina Hollender, Cristian Hofmann, Michael Deneke, and Bernhard Schmitz. 2010. Integrating cognitive load theory and concepts of human-computer interaction. Computer in Human Behavior 26, 6 (2010), 1278--1288.
[25]
Jukka Hyönä, Jorma Tommola, and Anna-Mari Alaja. 1995. Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks. The Quarterly Journal of Experimental Psychology 48, 3 (1995), 598--612.
[26]
B. Ismail and Anjum Khan. 2012. Image De-noising with a New Threshold Value Using Wavelets. Journal of Data Science 10 (2012), 259--270.
[27]
Robert J. K. Jacob and Keith S. Karn. 2003. Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises. In The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research, Jukka Hyönä, Ralph Radach, and Heiner Deubel (Eds.). Elsevier Science, Amsterdam, The Netherlands, 573--605.
[28]
Xianta Jiang, M. Stella Atkins, Geoffrey Tien, Roman Bednarik, and Bin Zheng. 2014. Pupil Responses During Discrete Goal-directed Movements. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, 2075--2084.
[29]
Xianta Jiang, Bin Zheng, Roman Bednarik, and M. Stella Atkins. 2015. Pupil responses to continuous aiming movements. International Journal of Human-Computer Studies 83 (2015), 1--11.
[30]
Marcel Adam Just and Patricia A. Carpenter. 1976. Eye Fixations and Cognitive Processes. Cognitive Psychology 8, 4 (October 1976), 441--480.
[31]
Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. Psychological Review 87, 4 (July 1980), 329--354.
[32]
Daniel Kahneman. 1973. Attention and effort. Prentice-Hall Englewood Cliffs, NJ.
[33]
Daniel Kahneman and Jackson Beatty. 1966. Pupillary Diameter and Load on Memory. Science 154 (1966), 1583--1585.
[34]
Peter Kiefer, Ioannis Giannopoulos, Andrew Duchowski, and Martin Raubal. 2016. Measuring Cognitive Load for Map Tasks through Pupil Diameter. In Proceedings of the Night International Conference on Geographic Information Science (GIScience 2016). Springer International Publishing.
[35]
Jeff Klingner, Rakshit Kumar, and Pat Hanrahan. 2008. Measuring the Task-Evoked Pupillary Response with a Remote Eye Tracker. In ETRA '08: Proceedings of the 2008 Symposium on Eye Tracking Research & Applications. ACM, New York, NY, 69--72.
[36]
Jeff Klingner, Barbara Tversky, and Pat Hanrahan. 2011. Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. Psychophysiology 48, 3 (2011), 323--332.
[37]
Krzysztof Krejtz, Andrew Duchowski, Izabela Krejtz, Agnieszka Szarkowska, and Agata Kopacz. 2016. Discerning Ambient/Focal Attention with Coefficient K. Transactions on Applied Perception 13, 3 (2016).
[38]
Jan-Louis Kruger, Esté Hefer, and Gordon Matthew. 2013. Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts. In Proceedings of the 2013 Conference on Eye Tracking South Africa (ETSA '13). ACM, New York, NY, 75--78.
[39]
LimeSurvey Project Team / Carsten Schmitz. 2015. LimeSurvey: An Open Source survey tool. LimeSurvey Project, Hamburg, Germany. http://www.limesurvey.org
[40]
Lisa M. Lix, Joanne C. Keselman, and H. J. Keselman. 1996. Consequences of Assumption Violations Revisited: A Quantitative Review of Alternatives to the One-Way Analysis of Variance F Test. Review of Educational Research 66, 4 (1996), 579--619.
[41]
Yongqiang Lyu, Xiaomin Luo, Jun Zhou, Chun Yu, Congcong Miao, Tong Wang, Yuanchun Shi, and Ken-ichi Kameyama. 2015. Measuring Photoplethysmogram-Based Stress-Induced Vascular Response Index to Assess Cognitive Load and Stress. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, 857--866.
[42]
Stephane Mallat and Wen Liang Hwang. 1992. Singularity Detection and Processing with Wavelets. IEEE Transactions on Information Theory 38, 2 (March 1992), 617--643.
[43]
Sandra P. Marshall. 2000. Method and Apparatus for Eye Tracking Monitoring Pupil Dilation to Evaluate Cognitive Activity. US Patent No. 6,090,051. (18 July 2000).
[44]
Sandra P. Marshall. 2002. The Index of Cognitive Activity: Measuring Cognitive Workload. In Proceedings of the 7th Human Factors Meeting. IEEE.
[45]
Sandra P. Marshall. 2007. Identifying Cognitive State from Eye Metrics. Aviation, Space, and Environmental Medicine 78, 5, Seciton II (May 2007), B165--B175(11). Supplement 1.
[46]
Ankit Mathur, Julia Gehrmann, and David A. Atchison. 2013. Pupil shape as viewed long the horizontal visual field. Journal of Vision 13, 6 (2013), 1--8.
[47]
Jum C. Nunnally and Ira H. Bernstein. 1994. Psychometric Theory (3rd ed.). McGraw-Hill Inc., New York, NY.
[48]
Sharon Oviatt. 2006. Human-centered Design Meets Cognitive Load Theory: Designing Interfaces That Help People Think. In Proceedings of the 14th ACM International Conference on Multimedia (MM '06). ACM, New York, NY, 871--880.
[49]
Fred Paas, Juhani E. Tuovinen, Huib Tabbers, and Pascal W. M. Van Gerven. 2003. Cognitive Load Measurement as a Means to Advance Cognitive Load Theory. Educational Psychologist 38, 1 (2003), 63--71.
[50]
Oskar Palinko and Andrew L. Kun. 2012. Exploring the Effects of Visual Cognitive Load and Illumination on Pupil Diameter in Driving Simulators. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, NY, 413--416.
[51]
Jonathan W Peirce. 2007. PsychoPy--Psychophysics Software in Python. Journal of neuroscience methods 162, 1 (2007), 8--13.
[52]
Vsevolod Peysakhovich. 2016. Study of pupil diameter and eye movements to enhance flight safety. Ph.D. Dissertation. Université de Toulouse, Toulouse, France.
[53]
Bastian Pfleging, Drea K. Fekety, Albrecht Schmidt, and Andrew L. Kun. 2016. A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, 5776--5788.
[54]
Tepring Piquado, Derek Isaacowitz, and Arthur Wingfield. 2010. Pupillometry as a Measure of Cognitive Effort in Younger and Older Adults. Psychophysiology 47, 3 (2010), 560--569.
[55]
R Development Core Team. 2011. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/ ISBN 3--900051-07-0.
[56]
Pallavi Raiturkar, Andrea Kleinsmith, Andreas Keil, Arunava Banerjee, and Eakta Jain. 2016. Decoupling Light Reflex from Pupillary Dilation to Measure Emotional Arousal in Videos. In Proceedings of the ACM Symposium on Applied Perception (SAP '16). ACM, New York, NY, 89--96.
[57]
Gergely Rakoczi, Andrew Duchowski, and Margit Pohl. 2014. Designing Online Tests for a Virtual Learning Environment: Evaluation of Visual Behaviour at Different Task Types. In Proceedings of the International Conference on Human Behavior in Design (HBiD). The Design Society, Ascona, Switzerland.
[58]
Emanuel Schmider, Matthias Ziegler, Erik Danay, Luzi Beyer, and Markus Bühner. 2010. Is It Really Robust? Reinvestigating the Robustness of ANOVA Against Violations of the Normal Distribution Assumption. Methodology 6, 4 (2010), 147--151.
[59]
Maximilian Schwalm. 2009. Pupillometrie als Methode zur Erfassung mentaler Beanspruchungen in automotiven Kontext. Ph.D. Dissertation. Universität des Saarlandes, Saarbrücken, Germany.
[60]
Yu Shi, Eric Choi, Ronnie Taib, and Fang Chen. 2009. Designing Cognition-Adaptive Human Computer Interface for Mission-Critical Systems. In Information Systems Development: Towards a Service Provision Society, George Angelos Papadopoulos, Wita Wojtkowski, Gregory Wojtkowski, Stanslaw Wrycza, and Joze Zupancic (Eds.). Springer Science+Business Media, LLC, New York, NY, 111--119.
[61]
Eva Siegenthaler, Francisco M. Costela, Michael B. McCamy, Leandro L. Di Stasi, Jorge Otero-Millan, Andreas Sonderegger, Rudolf Groner, Stephen Macknik, and Susana Martinez-Conde. 2014. Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes. European Journal of Neuroscience 39, 1 (2014), 1--8.
[62]
SR Research Ltd. 2008. EyeLink User Manual. SR Research Ltd., 5516 Main St., Osgoode, ON, Canada K0A 2W0. Version 1.4.0.
[63]
Lawrence Stark, Fergus W. Campbell, and John Atwood. 1958. Pupil Unrest: An Example of Noise in a Biological Servomechanism. Nature 182, 4639 (1958), 857--858.
[64]
John Sweller. 1976. The Effect of Task Complexity and Sequence on Rule Learning and Problem Solving. British Journal of Psychology 67, 4 (1976), 553--558.
[65]
John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257--285.
[66]
Jos N. van der Geest and Maarten A. Frens. 2002. Recording eye movements with video-oculography and scleral search coils: a direct comparison of two methods. 114, 2 (2002), 185--195.
[67]
Boris M. Velichkovsky, Markus Joos, Jens R. Helmert, and Sebastian Pannasch. 2005. Two Visual Systems and their Eye Movements: Evidence from Static and Dynamic Scene Perception. In CogSci 2005: Proceedings of the XXVII Conference of the Cognitive Science Society. 2283--2288.
[68]
Brani Vidakovi´ c and Peter Müller. 1991. Wavelets for kids: A tutorial introduction. Technical Report. Duke University.
[69]
Dong Wang, Fiona B. Mulvey, Jeff B. Pelz, and Kenneth Holmqvist. 2017. A study of artificial eyes for the measurement of precision in eye-trackers. Behavior Research Methods 49, 3 (2017), 947--959.
[70]
David L. Woods, Mark M. Kishiyama, E. William Yund, Timothy J. Herron, Ben Edwards, Oren Poliva, Robert F. Hink, and Bruce Reed. 2011. Improving digit span assessment of short-term verbal memory. Journal of Clinical and Experimental Neuropsychology 33, 1 (2011), 101--111.
[71]
Beste F. Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, Daniel Afergan, Remco Chang, and Robert J. K. Jacob. 2016. Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulgty based on Brain State. In Human Factors in Computing Systems: CHI '16 Conference Proceedings. ACM, San Jose, CA.

Cited By

View all

Index Terms

  1. The Index of Pupillary Activity: Measuring Cognitive Load vis-à-vis Task Difficulty with Pupil Oscillation

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
        April 2018
        8489 pages
        ISBN:9781450356206
        DOI:10.1145/3173574
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 21 April 2018

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. eye tracking
        2. pupillometry
        3. task difficulty

        Qualifiers

        • Research-article

        Funding Sources

        Conference

        CHI '18
        Sponsor:

        Acceptance Rates

        CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;
        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)1,059
        • Downloads (Last 6 weeks)94
        Reflects downloads up to 16 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Login options

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media