skip to main content
10.1145/3303772.3303775acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article

Counting Clicks is Not Enough: Validating a Theorized Model of Engagement in Learning Analytics

Published: 04 March 2019 Publication History

Abstract

Student engagement is often considered an overarching construct in educational research and practice. Though frequently employed in the learning analytics literature, engagement has been subjected to a variety of interpretations and there is little consensus regarding the very definition of the construct. This raises grave concerns with regards to construct validity: namely, do these varied metrics measure the same thing? To address such concerns, this paper proposes, quantifies, and validates a model of engagement which is both grounded in the theoretical literature and described by common metrics drawn from the field of learning analytics. To identify a latent variable structure in our data we used exploratory factor analysis and validated the derived model on a separate sub-sample of our data using confirmatory factor analysis. To analyze the associations between our latent variables and student outcomes, a structural equation model was fitted, and the validity of this model across different course settings was assessed using MIMIC modeling. Across different domains, the broad consistency of our model with the theoretical literature suggest a mechanism that may be used to inform both interventions and course design.

References

[1]
Panagiotis Adamopoulos. 2013. What makes a great MOOC? An interdisciplinary analysis of student retention in online courses. International Conference on Information Systems (ICIS 2013): Reshaping Society Through Information Systems Design 5 (01 2013), 4720--4740.
[2]
Oqab Alrashidi, Huy Phan, and Bing Ngu. 2016. Academic Engagement: An Overview of Its Definitions, Dimensions, and Major Conceptualisations. 9 (11 2016), 41--52.
[3]
James J. Appleton, Sandra L. Christenson, and Michael J. Furlong. 2008. Student engagement with school: Critical conceptual and methodological issues of the construct. Psychology in the Schools 45, 5 (1 5 2008), 369--386.
[4]
James J. Appleton, Sandra L. Christenson, Dongjin Kim, and Amy L. Reschly. 2006. Measuring cognitive and psychological engagement: Validation of the Student Engagement Instrument. Journal of School Psychology 44, 5 (1 10 2006), 427--445.
[5]
American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Joint Committee on Standards for Educational, and Psychological Testing. 2014. Standards for Educational and Psychological Testing.
[6]
Roger Azevedo. 2015. Defining and Measuring Engagement and Learning in Science: Conceptual, Theoretical, Methodological, and Analytical Issues. Educational Psychologist 50, 1 (2015), 84--94.
[7]
A. Alexander Beaujean. 2014. Latent variable modeling using R: A step-by-step guide. Routledge.
[8]
Natasha K. Bowen and Shenyang Guo. 2011. Structural Equation Modeling. Oxford University Press.
[9]
Timothy A. Brown. 2015. Methodology in the social sciences. Confirmatory factor analysis for applied research (2nd ed.). New York, NY, US: Guilford Press.
[10]
Robert C. MacCallum, Michael W. Browne, and Hazuki M. Sugawara. 1996. Power Analysis and Determination of Sample Size for Covariance Structure Modeling. 1 (06 1996), 130--149.
[11]
Sandra L. Christenson, Amy L. Reschly, James J. Appleton, Sarah Berman, Deanna M. Spanjers, and P. Varro. 2008. Best practices in fostering student engagement. National Association of School Psychologists, 1099--1120.
[12]
Sandra L. Christenson and Amy R. Anderson. 2002. Commentary: The centrality of the learning context for students' academic enabler skills. School Psychology Review 31 (2002), 378--393.
[13]
Thomas D Cook and Donald T Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin.
[14]
Anna B. Costello and Jason W. Osborne. 2005. Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most From Your Analysis. 10 (01 2005), 1--9.
[15]
Jennifer DeBoer, Andrew D. Ho, Glenda S. Stump, and Lori Breslow. 2014. Changing 'course': Reconceptualizing educational variables for massive open online courses. Educational Researcher 43, 2 (2014), 74--94.
[16]
Nia Dowell, Oleksandra Skrypnyk, Srećko Joksimović, Arthur C. Graesser, Shane Dawson, Dragan Gašević, Pieter de Vries, Thieme Hennis, and Vitomir Kovanović. 2015. Modeling Learners' Social Centrality and Performance through Language and Discourse. (2015). https://eric.ed.gov/?id=ED560532 presented at the Proceedings of the 8th International Conference on Educational Data Mining.
[17]
Leandre Fabrigar, Duane Wegener, Robert C. MacCallum, and Erin J. Strahan. 1999. Evaluating the use of Exploratory Factor Analysis in psychological research. 4 (01 1999), 272--299.
[18]
Andy Field, Jeremy Miles, and Zoe Field. 2012. Discovering Statistics Using R. SAGE Publications.
[19]
Jeremy D. Finn. 1989. Withdrawing From School. Review of Educational Research 59, 2 (1989), 117--142.
[20]
Charles W. Fisher and David C. Berliner. 1985. Perspectives on instructional time. New York: Longman.
[21]
Jennifer A Fredricks, Phyllis C Blumenfeld, and Alison H Paris. 2004. School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research 74, 1 (2004), 59--109.
[22]
Dragan Gašević, Shane Dawson, and George Siemens. 2015. Let's not forget: Learning analytics are about learning. TechTrends 59, 1 (Jan 2015), 64--71.
[23]
Dragan Gašević, Jelena Jovanović, Abelardo Pardo, and Shane Dawson. 2017. Detecting Learning Strategies with Analytics: Links with Self-reported Measures and Academic Performance. Journal of Learning Analytics 4, 2 (2017).
[24]
Peter Goodyear. 2002. Psychological Foundations for Networked Learning. Springer London, London, 49--75.
[25]
Arthur C. Graesser, Danielle S. McNamara, and Jonna M. Kulikowich. 2011. Coh-Metrix: Providing Multilevel Analyses of Text Characteristics. Educational Researcher 40, 5 (2011), 223--234. http://www.jstor.org/stable/41238943
[26]
Christian Gütl, Vanessa Rizzardini, Rocael Hernándezand Chang, and Miguel Morales. 2014. Attrition in MOOC: Lessons Learned from Drop-Out Students. In Learning Technology for Education in Cloud. MOOC and Big Data, Lorna Uden, Jane Sinclair, Yu-Hui Tao, and Dario Liberona (Eds.). Springer International Publishing, Cham, 37--48.
[27]
James C. Hayton, David G. Allen, and Vida Scarpello. 2004. Factor Retention Decisions in Exploratory Factor Analysis: a Tutorial on Parallel Analysis. Organizational Research Methods 7, 2 (2004), 191--205.
[28]
John L. Horn and John J. Mcardle. 1992. A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research 18, 3 (1992), 117--144.
[29]
Li-tze Hu and Peter M. Bentler. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal 6, 1 (1999), 1--55.
[30]
IBM Tone Analyzer. n.d. https://www.ibm.com/watson/developercloud/tone-analyzer.html.
[31]
Srećko Joksimović, Nia Dowell, Oleksandra Skrypnyk, Vitomir Kovanović, Dragan Gašević, Shane Dawson, and Arthur C. Graesser. 2015. How Do You Connect?: Analysis of Social Capital Accumulation in Connectivist MOOCs. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (LAK '15). ACM, New York, NY, USA, 64--68.
[32]
Srećko Joksimović, Oleksandra Poquet, Vitomir Kovanović, Nia Dowell, Caitlin Mills, Dragan Gašević, Shane Dawson, Arthur C. Graesser, and Christopher Brooks. 2018. How Do We Model Learning at Scale? A Systematic Review of Research on MOOCs. Review of Educational Research 88, 1 (2018), 43--86.
[33]
Chris Jones. 2008. Networked Learning - A social practice perspective. In Proceedings of the 6th International Conference on Networked Learning. 616--623. http://oro.open.ac.uk/20659/
[34]
Karl Jöreskog and Arthur S. Goldberger. 1975. Estimation of a Model with Multiple Indicators and Multiple Causes of a Single Latent Variable. 70 (1975), 631--639.
[35]
Karl G Jöreskog. 1999, June 22. How Large Can a Standardized Coefficient be? http://www.ssicentral.com/lisrel/techdocs/HowLargeCanaStandardizedCoefficientbe.pdf
[36]
Michael T. Kane. 2013. Validating the interpretations and uses of test scores. Journal of Educational Measurement 50, 1 (2013), 1--73.
[37]
David Kaplan. 2009. Structural Equation Modeling (2nd ed.): Foundations and Extensions. SAGE Publications.
[38]
Eun Sook Kim, Myeongsun Yoon, and Taehun Lee. 2012. Testing Measurement Invariance Using MIMIC: Likelihood Ratio Test With a Critical Value Adjustment. Educational and Psychological Measurement 72, 3 (2012), 469--492.
[39]
René Kizilcec, Chris Piech, and Emily Schneider. 2013. Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In Proceedings of the Third International Conference on Learning Analytics & Knowledge (LAK '13). ACM, 170--179.
[40]
Vitomir Kovanović, Dragan Gašević, Shane Dawson, Srećko Joksimović, and Ryan Baker. 2016. Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings. Journal of Learning Analytics 2, 3 (2016), 81--110.
[41]
Nan Li, Lukasz Kidziński, Patrick Jermann, and Pierre Dillenbourg. 2015. MOOC video interaction patterns: What do they tell us? Lecture Notes in Computer Science, Vol. 9307. Springer, Berlin, Germany, 197--210.
[42]
Jason Lodge and Melinda Lewis. 2012. Pigeon pecks and mouse clicks: Putting the learning back into learning analytics. In ASCILITE - Australian Society for Computers in Learning in Tertiary Education Annual Conference 2012, M. Brown, M. Hartnett, and T. Stewart(Eds.). Australasian Society for Computers in Learning in Tertiary Education. https://www.learntechlib.org/p/42645
[43]
Jane Loevinger. 1957. Objective Tests as Instruments of Psychological Theory. Psychological Reports 3, 3 (1957), 635--694.
[44]
Herbert W. Marsh, Kit-Tai Hau, and Zhonglin Wen. 2004. In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings. Structural Equation Modeling: A Multidisciplinary Journal 11, 3 (2004), 320--341.
[45]
Danielle S. McNamara, Arthur C. Graesser, Philip M. McCarthy, and Zhiqiang Cai. 2014. Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge University Press.
[46]
Samuel Messick. 1995. Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice (1995), 5--8.
[47]
Samuel Messick. 1995. Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American psychologist 50, 9 (1995), 741.
[48]
Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume, III, and Lise Getoor. 2014. Uncovering Hidden Engagement Patterns for Predicting Learner Performance in MOOCs. In Proceedings of the First ACM Conference on Learning @ Scale Conference (L@S '14). ACM, New York, NY, USA, 157--158.
[49]
Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume III, and Lise Getoor. 2014. Learning Latent Engagement Patterns of Students in Online Courses. (2014). https://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8571
[50]
Tenko Raykov. 1997. Estimation of Composite Reliability for Congeneric Measures. Applied Psychological Measurement 21, 2 (1997), 173--184.
[51]
Justin Reich. 2015. Rebooting MOOC Research. Science 347, 6217 (2015), 34--35.
[52]
Amy L. Reschly and Sandra L. Christenson. 2012. Jingle, Jangle, and Conceptual Haziness: Evolution and Future Directions of the Engagement Construct. Springer US, Boston, MA, 3--19.
[53]
William Revelle. 2018. psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. https://CRAN.R-project.org/package=psych R package version 1.8.4.
[54]
Tim Rogers, Dragan Gašević, and Shane Dawson. 2016. Learning analytics and the imperative for theory driven research. In The SAGE Handbook of E-Learning Research, 2nd edition, Caroline Haythornthwaite, Richard Andrews, Jude Fransma, and Eric Meyers (Eds.). SAGE Publications Ltd, London, UK, 232--250.
[55]
Yves Rosseel. 2012. lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software 48, 2 (2012), 1--36. http://www.jstatsoft.org/v48/i02/
[56]
Jose Luis Santos, Joris Klerkx, Erik Duval, David Gago, and Luis Rodríguez. 2014. Success, Activity and Drop-outs in MOOCs an Exploratory Study on the UNED COMA Courses. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (LAK '14). ACM, New York, NY, USA, 98--102.
[57]
Albert H. Segars. 1997. Assessing the unidimensionality of measurement: a paradigm and illustration within the context of information systems research. Omega 25, 1 (1997), 107--121.
[58]
Lee S. Shulman. 1970. Reconstruction of Educational Research. Review of Educational Research 40, 3 (1970), 371--396.
[59]
Tanmay Sinha and Justine Cassell. 2015. Connecting the Dots: Predicting Student Grade Sequences from Bursty MOOC Interactions over Time. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (L@S '15). ACM, New York, NY, USA, 249--252.
[60]
Conrad Tucker, Barton K. Pursel, and Aanna Divinsky. 2014. Mining studentgenerated textual data in MOOCS and quantifying their effects on student performance and learning outcomes. (2014). https://peer.asee.org/22840 Paper presented at ASEE Annual Conference, Indianapolis, Indiana.
[61]
Duy Vu, Philippa Pattison, and Garry Robins. 2015. Relational event models for social learning in MOOCs. Social Networks 43 (2015), 121--135.
[62]
Sukaina Walji, Andrew Deacon, Janet Small, and Laura Czerniewicz. 2016. Learning through engagement: MOOCs as an emergent form of provision. Distance Education 37, 2 (2016), 208--223. arXiv:https://doi.org/10.1080/01587919.2016.1184400
[63]
Xu Wang, Diyi Yang, Miaomiao Wen, Kenneth R. Koedinger, and Carolyn Penstein Rosé. 2015. Investigating How Student's Cognitive Behavior in MOOC Discussion Forum Affect Learning Gains. In Proceedings of the 8th International Conference on Educational Data Mining. 226--233.
[64]
Jacob Whitehill, Joseph Williams, Glenn Lopez, Cody Austun Coleman, and Justin Reich. 2015. Beyond Prediction: First Steps Toward Automatic Intervention in MOOC Student Stopout. In Proceedings of the 8th International Conference on Educational Data Mining (EDM'15). 171--178.
[65]
Philip H Winne and Dianne Jamieson-Noel. 2002. Exploring students' calibration of self reports about study tactics and achievement. Contemporary Educational Psychology 27, 4 (2002), 551--572.
[66]
Alyssa Friend Wise and David Shaffer. 2015. Why theory matters more than ever in the age of big data. 2 (01 2015), 5--13.
[67]
Cheng Ye, John S. Kinnebrew, Gautam Biswas, Brent J. Evans, Douglas H. Fisher, Gayathri Narasimham, and Katherine A. Brady. 2015. Behavior Prediction in MOOCs Using Higher Granularity Temporal Information. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (L@S '15). ACM, New York, NY, USA, 335--338.
[68]
Mingming Zhou and Philip H. Winne. 2012. Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction 22, 6 (2012), 413--419.

Cited By

View all
  • (2024)Contract Cost Management and Supply Chain Performance of County Referral Hospitals in KenyaInternational Journal of Social Science and Humanities Research (IJSSHR) ISSN 2959-7056 (o); 2959-7048 (p)10.61108/ijsshr.v2i2.1112:2Online publication date: 30-Jun-2024
  • (2024)Feedback mit Learning Analytics – Interdisziplinäres Design eines Dashboards für StudierendeFeedback with learning analytics – Interdisciplinary design of a dashboard for studentsZeitschrift für Hochschulentwicklung10.21240/zfhe/19-4/0519:4(77-94)Online publication date: 22-Dec-2024
  • (2024)Unpacking student engagement in higher education learning analytics: a systematic reviewInternational Journal of Educational Technology in Higher Education10.1186/s41239-024-00493-y21:1Online publication date: 20-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
LAK19: Proceedings of the 9th International Conference on Learning Analytics & Knowledge
March 2019
565 pages
ISBN:9781450362566
DOI:10.1145/3303772
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 March 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Engagement
  2. Factor Analysis
  3. MOOCs
  4. Measurement Invariance
  5. Structural Equation Modeling

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

LAK19

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)106
  • Downloads (Last 6 weeks)3
Reflects downloads up to 02 Feb 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media