skip to main content
10.1145/3517031.3529631acmconferencesArticle/Chapter ViewAbstractPublication PagesetraConference Proceedingsconference-collections
research-article

LSTMs can distinguish dental expert saccade behavior with high ”plaque-urracy”

Published: 08 June 2022 Publication History

Abstract

Much of the current expertise literature has found that domain specific tasks evoke different eye movements. However, research has yet to predict optimal image exploration using saccadic information and to identify and quantify differences in the search strategies between learners, intermediates, and expert practitioners. By employing LSTMs for scanpath classification, we found saccade features over time could distinguish all groups at high accuracy. The most distinguishing features were saccade velocity peak (72%), length (70%), and velocity average (68%). These findings promote the holistic theory of expert visual exploration that experts can quickly process the whole scene using longer and more rapid saccade behavior initially. The potential to integrate expertise model development from saccadic scanpath features into intelligent tutoring systems is the ultimate inspiration for our research. Additionally, this model is not confined to visual exploration in dental xrays, rather it can extend to other medical domains.

References

[1]
Modi Owied Al-Moteri, Mark Symmons, Virginia Plummer, and Simon Cooper. 2017. Eye tracking to investigate cue processing in medical decision-making: A scoping review. Computers in Human Behavior 66 (2017), 52–66.
[2]
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine 34, 6 (2017), 26–38.
[3]
Dan Assaf, Eyal Amar, Norbert Marwan, Yair Neuman, Moshe Salai, and Ehud Rath. 2016. Dynamic patterns of expertise: the case of orthopedic medical diagnosis. Plos One 11, 7 (2016), e0158820.
[4]
Stephanie Brams, Gal Ziv, Ignace TC Hooge, Oron Levin, Thomas De Brouwere, Johny Verschakelen, Siska Dauwe, A Mark Williams, Johan Wagemans, and Werner F Helsen. 2020. Focal lung pathology detection in radiology: Is there an effect of experience on visual search behavior?Attention, Perception, & Psychophysics(2020), 1–14.
[5]
Stephanie Brams, Gal Ziv, Oron Levin, Jochim Spitz, Johan Wagemans, A Mark Williams, and Werner F Helsen. 2019. The relationship between gaze behavior, expertise, and performance: A systematic review.Psychological Bulletin 145, 10 (2019), 980.
[6]
Tad T Brunyé, Trafton Drew, Manob Jyoti Saikia, Kathleen F Kerr, Megan M Eguchi, Annie C Lee, Caitlin May, David E Elder, and Joann G Elmore. 2021. Melanoma in the blink of an eye: Pathologists’ rapid detection, classification, and localization of skin abnormalities. Visual Cognition (2021), 1–15.
[7]
Guy Thomas Buswell. 1935. How people look at pictures: a study of the psychology and perception in art.Univ. Chicago Press.
[8]
Nora Castner, Enkelejda Kasneci, Thomas Kübler, Katharina Scheiter, Juliane Richter, Thérése Eder, Fabian Hüttig, and Constanze Keutel. 2018a. Scanpath comparison in medical image reading skills of dental students: distinguishing stages of expertise development. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications. ACM, 39.
[9]
Nora Castner, Solveig Klepper, Lena Kopnarski, Fabian Hüttig, Constanze Keutel, Katharina Scheiter, Juliane Richter, Thérése Eder, and Enkelejda Kasneci. 2018b. Overlooking: the nature of gaze behavior and anomaly detection in expert dentists. In Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data. ACM, 8.
[10]
Nora Castner, Thomas C Kuebler, Katharina Scheiter, Juliane Richter, Therese Eder, Fabian Huettig, Constanze Keutel, and Enkelejda Kasneci. 2020. Deep Semantic Gaze Embedding and Scanpath Comparison for Expertise Classification during OPT Viewing. In ACM Symposium on Eye Tracking Research and Applications (Stuttgart, Germany) (ETRA ’20 Full Papers). Association for Computing Machinery, New York, NY, USA, Article 18, 10 pages. https://doi.org/10.1145/3379155.3391320
[11]
Michelene TH Chi. 2006. Two approaches to the study of expert’ characteristics. Cambridge University Press, 21–30.
[12]
Lindsey Cooper, Alastair G Gale, Janak Saada, Swamy Gedela, Hazel J Scott, and Andoni Toms. 2010. The assessment of stroke multidimensional CT and MR imaging using eye movement analysis: does modality preference enhance observer performance?7627 (2010), 76270B.
[13]
David P Crabb, Nicholas D Smith, and Haogang Zhu. 2014. What’s on TV? Detecting age-related neurodegenerative eye disease using eye movement scanpaths. Frontiers in Aging Neuroscience 6 (2014), 312.
[14]
Emily M Crowe, Iain D Gilchrist, and Christopher Kent. 2018. New approaches to the analysis of eye movement behaviour across expertise while viewing brain MRIs. Cognitive Research: Principles and Implications 3, 1(2018), 12.
[15]
Alan Davies, Markel Vigo, Simon Harper, and Caroline Jay. 2018. Using simultaneous scanpath visualization to investigate the influence of visual behaviour on medical image interpretation. Journal of Eye Movement Research 10, 5 (Feb. 2018). https://doi.org/10.16910/jemr.10.5.11
[16]
Tim Donovan and Damien Litchfield. 2013. Looking for cancer: Expertise related differences in searching and decision making. Applied Cognitive Psychology 27, 1 (2013), 43–49.
[17]
Trafton Drew, Melissa Le-Hoa Vo, Alex Olwal, Francine Jacobson, Steven E Seltzer, and Jeremy M Wolfe. 2013. Scanners and drillers: Characterizing expert visual search through volumetric images. Journal of Vision 13, 10 (2013), 3–3.
[18]
Hubert L Dreyfus and Stuart E Dreyfus. 1986. The power of human intuition and expertise in the era of the computer. Mind Over Machine. Nueva York: the Free Press (1986).
[19]
K Anders Ericsson and Andreas C Lehmann. 1996. Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology 47, 1 (1996), 273–305.
[20]
Karla K Evans, Diane Georgian-Smith, Rosemary Tambouret, Robyn L Birdwell, and Jeremy M Wolfe. 2013. The gist of the abnormal: Above-chance medical decision making in the blink of an eye. Psychonomic Bulletin & Review 20, 6 (2013), 1170–1175.
[21]
Paul J Feltovich, Michael J Prietula, and K Anders Ericsson. 2006. Studies of expertise from psychological perspectives.In The Cambridge Handbook of Expertise and Expert Performance, K Anders Ericsson, Robert R Hoffman, and Aaron Kozbelt (Eds.). Cambridge University Press, 41–67.
[22]
Sharon E Fox and Beverly E Faulkner-Jones. 2017. Eye-Tracking in the Study of Visual Expertise: Methodology and Approaches in Medicine.Frontline Learning Research 5, 3 (2017), 29–40.
[23]
Robert M French, Yannick Glady, and Jean-Pierre Thibaut. 2017. An evaluation of scanpath-comparison and machine-learning classification algorithms used to study the dynamics of analogy making. Behavior Research Methods 49, 4 (2017), 1291–1302.
[24]
Wolfgang Fuhl, Nora Castner, Thomas Kübler, Alexander Lotz, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2019. Ferns for Area of Interest Free Scanpath Classification. In Proceedings of the ACM Symposium on Eye Tracking Research & Applications (Denver, Colorado) (ETRA ’19). Association for Computing Machinery, New York, NY, USA, Article 39, 5 pages. https://doi.org/10.1145/3314111.3319826
[25]
Ziba Gandomkar and Claudia Mello-Thoms. 2019. Visual search in breast imaging. the British Journal of Radiology 92, 1102 (2019), 20190057.
[26]
Ziba Gandomkar, Kevin Tay, Patrick C Brennan, and Claudia Mello-Thoms. 2017. A model based on temporal dynamics of fixations for distinguishing expert radiologists’ scanpaths. In Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, Vol. 10136. International Society for Optics and Photonics, 1013606.
[27]
Ziba Gandomkar, Kevin Tay, Patrick C Brennan, and Claudia Mello-Thoms. 2018. Recurrence quantification analysis of radiologists’ scanpaths when interpreting mammograms. Medical Physics 45, 7 (2018), 3052–3062.
[28]
Aarthi Ganesan, Maram Alakhras, Patrick C Brennan, and Claudia Mello-Thoms. 2018. A review of factors influencing radiologists’ visual search behaviour. Journal of Medical Imaging and Radiation Oncology 62, 6 (2018), 747–757.
[29]
Andreas Gegenfurtner, Erno Lehtinen, and Roger Säljö. 2011. Expertise differences in the comprehension of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational Psychology Review 23, 4 (2011), 523–552.
[30]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning, Chapter 10: Sequence Modeling: Recurrent and Recursive Nets. MIT Press. http://www.deeplearningbook.org.
[31]
Thorsten Grünheid, Dustin A Hollevoet, James R Miller, and Brent E Larson. 2013. Visual scan behavior of new and experienced clinicians assessing panoramic radiographs. Journal of the World Federation of Orthodontists 2, 1(2013), e3–e7.
[32]
Brian P Hermanson, Grant C Burgdorf, John F Hatton, Darrin M Speegle, and Karl F Woodmansey. 2018. Visual fixation and scan patterns of dentists viewing dental periapical radiographs: an eye tracking pilot study. Journal of Endodontics 44, 5 (2018), 722–727.
[33]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.
[34]
Benedikt Hosp, Myat Su Yin, Peter Haddawy, Ratthapoom Watcharopas, Paphon Sa-ngasoongsong, and Enkelejda Kasneci. 2021. Differentiating Surgeons’ Expertise solely by Eye Movement Features. In Companion Publication of the 2021 International Conference on Multimodal Interaction. 371–375.
[35]
Chartene H Hu, Harold L Kundel, Calvin F Nodine, Elizabeth A Krupinski, and Lawrence C Toto. 1994. Searching for bone fractures: a comparison with pulmonary nodule search. Academic Radiology 1, 1 (1994), 25–32.
[36]
Elizabeth L Irving, Martin J Steinbach, Linda Lillakas, Raiju J Babu, and Natalie Hutchings. 2006. Horizontal saccade dynamics across the human life span. Investigative ophthalmology & visual science 47, 6 (2006), 2478–2484.
[37]
Halszka Jarodzka, Kenneth Holmqvist, and Marcus Nyström. 2010. A vector-based, multidimensional scanpath similarity measure. In Proceedings of the 2010 symposium on eye-tracking research & applications. ACM, 211–218.
[38]
Conor Kelton, Zijun Wei, Seoyoung Ahn, Aruna Balasubramanian, Samir R Das, Dimitris Samaras, and Gregory Zelinsky. 2019. Reading detection in real-time. In Proceedings of the ACM Symposium on Eye Tracking Research & Applications. 1–5.
[39]
Naoko Koide, Takatomi Kubo, Satoshi Nishida, Tomohiro Shibata, and Kazushi Ikeda. 2015. Art expertise reduces influence of visual salience on fixation in viewing abstract-paintings. Plos One 10, 2 (2015), e0117696.
[40]
Ellen M Kok, Anique BH De Bruin, Simon GF Robben, and Jeroen JG Van Merrienboer. 2012. Looking in the same manner but seeing it differently: Bottom-up and expertise effects in radiology. Applied Cognitive Psychology 26, 6 (2012), 854–862.
[41]
Ellen M Kok, Halszka Jarodzka, Anique BH de Bruin, Hussain AN BinAmir, Simon GF Robben, and Jeroen JG van Merriënboer. 2016. Systematic viewing in radiology: seeing more, missing less?Advances in Health Sciences Education 21, 1 (2016), 189–205.
[42]
Elizabeth A Krupinski. 1996. Visual scanning patterns of radiologists searching mammograms. Academic Radiology 3, 2 (1996), 137–144.
[43]
Elizabeth A Krupinski, Allison A Tillack, Lynne Richter, Jeffrey T Henderson, Achyut K Bhattacharyya, Katherine M Scott, Anna R Graham, Michael R Descour, John R Davis, and Ronald S Weinstein. 2006. Eye-movement study and human performance using telepathology virtual slides. Implications for medical education and differences with experience. Human Pathology 37, 12 (2006), 1543–1556.
[44]
Thomas C Kübler and Enkelejda Kasneci. 2015. Automated Comparison of Scanpaths in Dynamic Scenes. In SAGA-International Workshop on Solutions for Automatic Gaze Data Analysis: Proceedings. 1–3.
[45]
Thomas C Kübler, Colleen Rothe, Ulrich Schiefer, Wolfgang Rosenstiel, and Enkelejda Kasneci. 2017. SubsMatch 2.0: Scanpath comparison and classification based on subsequence frequencies. Behavior research methods 49, 3 (2017), 1048–1064.
[46]
Harold L Kundel, Calvin F Nodine, Emily F Conant, and Susan P Weinstein. 2007. Holistic component of image perception in mammogram interpretation: gaze-tracking study. Radiology 242, 2 (2007), 396–402.
[47]
Kai Kunze, Yuzuko Utsumi, Yuki Shiga, Koichi Kise, and Andreas Bulling. 2013. I know what you are reading: recognition of document types using mobile eye tracking. In Proceedings of the 2013 International Symposium on Wearable Computers. 113–116.
[48]
Rui Li, Jeff Pelz, Pengcheng Shi, Cecilia Ovesdotter Alm, and Anne R Haake. 2012. Learning eye movement patterns for characterization of perceptual expertise. In Proceedings of the ACM Symposium on Eye Tracking Research & Applications. 393–396.
[49]
Rui Li, Pengcheng Shi, Jeff Pelz, Cecilia O Alm, and Anne R Haake. 2016. Modeling eye movement patterns to characterize perceptual skill in image-based diagnostic reasoning processes. Computer Vision and Image Understanding 151 (2016), 138–152.
[50]
Yu Li, Carla Allen, and Chi-Ren Shyu. 2019. Quantifying and understanding the differences in visual activities with contrast subsequences. In Proceedings of the ACM Symposium on Eye Tracking Research & Applications. 1–5.
[51]
Susan Mallett, Peter Phillips, Thomas R Fanshawe, Emma Helbren, Darren Boone, Alastair Gale, Stuart A Taylor, David Manning, Douglas G Altman, and Steve Halligan. 2014. Tracking eye gaze during interpretation of endoluminal three-dimensional CT colonography: visual perception of experienced and inexperienced readers. Radiology 273, 3 (2014), 783–792.
[52]
David Manning, Susan Ethell, Tim Donovan, and Trevor Crawford. 2006. How do radiologists do it? The influence of experience and training on searching for chest nodules. Radiography 12, 2 (2006), 134–142.
[53]
Ezgi Mercan, Linda G Shapiro, Tad T Brunyé, Donald L Weaver, and Joann G Elmore. 2018. Characterizing diagnostic search patterns in digital breast pathology: Scanners and drillers. Journal of Digital Imaging 31, 1 (2018), 32–41.
[54]
DP Munoz, JR Broughton, JE Goldring, and IT Armstrong. 1998. Age-related performance of human subjects on saccadic eye movement tasks. Experimental brain research 121, 4 (1998), 391–400.
[55]
DOUGLAS P Munoz, DAVID M Waitzman, and ROBERT H Wurtz. 1996. Activity of neurons in monkey superior colliculus during interrupted saccades. Journal of Neurophysiology 75, 6 (1996), 2562–2580.
[56]
Calvin F Nodine, Harold L Kundel, Sherri C Lauver, and Lawrence C Toto. 1996. Nature of expertise in searching mammograms for breast masses. Academic Radiology 3, 12 (1996), 1000–1006.
[57]
Calvin F Nodine and Claudia Mello-Thoms. 2000. The nature of expertise in radiology. Handbook of Medical Imaging. SPIE(2000), 859–895.
[58]
Sebastian Pannasch, Jens R Helmert, Katharina Roth, Ann-Katrin Herbold, and Henrik Walter. 2008. Visual fixation durations and saccade amplitudes: Shifting relationship in a variety of conditions. Journal of Eye Movement Research 2, 2 (2008).
[59]
Derrick Parkhurst, Klinton Law, and Ernst Niebur. 2002. Modeling the role of salience in the allocation of overt visual attention. Vision Research 42, 1 (2002), 107–123.
[60]
Michael Polanyi. 1962. Personal knowledge: towards a post-critical. Philosophy (1962).
[61]
SensoMotoric Instruments 2017. BeGaze Manual(version 3.7 ed.). SensoMotoric Instruments.
[62]
James Shanteau. 1992. Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes 53, 2 (1992), 252–266.
[63]
Hongxin Shao and Boon-Hee Soong. 2016. Traffic flow prediction with long short-term memory networks (LSTMs). In 2016 IEEE Region 10 Conference (TENCON). IEEE, 2986–2989.
[64]
Komi Sodoké, Roger Nkambou, Aude Dufresne, and Issam Tanoubi. 2020. Toward a deep convolutional LSTM for eye gaze spatiotemporal data sequence classification. In Proceedings of the 13th International Conference on Educational Data Mining. 672–676.
[65]
SB Stevenson, FC Volkmann, JP Kelly, and Lorrin A Riggs. 1986. Dependence of visual suppression on the amplitudes of saccades and blinks. Vision research 26, 11 (1986), 1815–1824.
[66]
Y. Tao and M. Shyu. 2019. SP-ASDNet: CNN-LSTM Based ASD Classification Model using Observer ScanPaths. In 2019 IEEE International Conference on Multimedia Expo Workshops (ICMEW). 641–646. https://doi.org/10.1109/ICMEW.2019.00124
[67]
Daniel P Turgeon and Ernest WN Lam. 2016. Influence of experience and training on dental students’ examination performance regarding panoramic images. Journal of dental education 80, 2 (2016), 156–164.
[68]
Munenori Uemura, Morimasa Tomikawa, Ryuichi Kumashiro, Tiejun Miao, Ryota Souzaki, Satoshi Ieiri, Kenoki Ohuchida, Alan T Lefor, and Makoto Hashizume. 2014. Analysis of hand motion differentiates expert and novice surgeons. Journal of Surgical Research 188, 1 (2014), 8–13.
[69]
A Van der Gijp, CJ Ravesloot, H Jarodzka, MF van der Schaaf, IC van der Schaaf, Jan PJ van Schaik, and Th J Ten Cate. 2017. How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education 22, 3 (2017), 765–787.
[70]
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and S Yu Philip. 2017. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. In Advances in Neural Information Processing Systems. 879–888.
[71]
Amy L Warren, Tyrone L Donnon, Catherine R Wagg, Heather Priest, and Nicole J Fernandez. 2018. Quantifying Novice and Expert Differences in Visual Diagnostic Reasoning in Veterinary Pathology Using Eye-Tracking Technology. Journal of Veterinary Medical Education 45, 3 (2018), 295–306.
[72]
Jeremy M Wolfe, Chia-Chien Wu, Jonathan Li, and Sneha B Suresh. 2021. What do experts look at and what do experts find when reading mammograms?Journal of Medical Imaging 8, 4 (2021), 045501.
[73]
Greg Wood, Karen M Knapp, Benjamin Rock, Chris Cousens, Carl Roobottom, and Mark R Wilson. 2013. Visual expertise in detecting and diagnosing skeletal fractures. Skeletal Radiology 42, 2 (2013), 165–172.
[74]
Myat Su Yin, Peter Haddawy, Benedikt Hosp, Paphon Sa-ngasoongsong, Thanwarat Tanprathumwong, Madereen Sayo, Supawit Yangyuenpradorn, and Akara Supratak. 2020. A Study of Expert/Novice Perception in Arthroscopic Shoulder Surgery. In Proceedings of the 4th International Conference on Medical and Health Informatics. 71–77.
[75]
Hong-Jun Yoon, Folami Alamudun, Kathy Hudson, Garnetta Morin-Ducote, and Georgia Tourassi. 2018. Deep gaze velocity analysis during mammographic reading for biometric identification of radiologists. Journal of Human Performance in Extreme Environments 14, 1(2018), 3.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ETRA '22: 2022 Symposium on Eye Tracking Research and Applications
June 2022
408 pages
ISBN:9781450392525
DOI:10.1145/3517031
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deep Learning
  2. Expertise
  3. Eye Tracking
  4. Medical image interpretation
  5. Scanpath analysis
  6. intelligent tutoring systems

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • Machine Learning Cluster of Excellence, EXC number 2064/1

Conference

ETRA '22

Acceptance Rates

ETRA '22 Paper Acceptance Rate 15 of 39 submissions, 38%;
Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)20
  • Downloads (Last 6 weeks)3
Reflects downloads up to 27 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media