skip to main content
10.1145/2855321.2855336acmotherconferencesArticle/Chapter ViewAbstractPublication PageseuroplopConference Proceedingsconference-collections
research-article

Data-driven design pattern production: a case study on the ASSISTments online learning system

Published: 08 July 2015 Publication History

Abstract

Recently, online learning systems such as cognitive tutors, online courses, and massive open online courses (MOOCS) increased in popularity in various domains. The design quality of online learning systems is difficult to maintain. Multiple stakeholders are involved (e.g., software developers, interaction designers, learning scientists, teachers), the system is complex, there are rapid changes in software, platforms (e.g., mobile, tablet, desktop) and learning subject content, and so forth. Many existing online learning systems collect a significant amount of data that describe students' learning gains and affective states, which are indirect measures of system quality. Analysis of online learning system data can uncover linkages between particular design choices made and student learning. In this paper, we describe the data-driven design pattern production (3D2P) methodology to prospect, mine, write, and evaluate design patterns for online learning systems from collected data. Design patterns are high quality solutions for recurring problems in a particular context. Patterns identified with 3D2P can guide the addition of new content and the modification of system designs to maintain the quality of online learning systems. A case study on the ASSISTments math online learning system is presented to demonstrate the 3D2P methodology and discuss its benefits and limitations.

References

[1]
Andrew Arnold, Richard Scheines, Joseph Beck, and Bill Jerome. 2005. Time and attention: Students, sessions, and tasks. In Proceedings of the AAAI 2005 Workshop Educational Data Mining, 62--66.
[2]
ASSISTments. n.d. Retrieved February 1, 2015 from https://www.assistments.org/
[3]
Cristopher Alexander, Sara Ishikawa, and Murray Silverstein. 1977. A pattern language: towns, buildings, construction. Vol. 2. Oxford University Press, New York, NY.
[4]
Elaine Allen and Jeff Seaman. 2008. Staying the course: Online education in the United States. Sloan Consortium, Needham, MA.
[5]
Júnia Coutinho Anacleto, Américo Talarico Neto, and Vânia Paula de Almeida Néris. 2009. Cog-Learn: An e-Learning Pattern Language for Web-based Learning Design. eLearn Magazine, 8 (2009), ACM, New York, NY.
[6]
Dana L. Anthony. 1996. Patterns for classroom education. In Pattern Languages of Program Design 2, John Vlissides, James Coplien, and Norman Kerth (Eds.). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 391--406.
[7]
Anthony R. Artino and Kenneth D. Jones. 2012. Exploring the complex relations between achievement emotions and self-regulated learning behaviors in online learning. The Internet and Higher Education 15, 3, 170--175.
[8]
Ryan S. J. D. Baker, Sujith M. Gowda, Michael Wixon, Jessica Kalka, Angela Z. Wagner, Aatish Salvi, Vincent Aleven, Gail W. Kusbit, Jaclyn Ocumpaugh, and Lisa Rossi. 2012. Towards Sensor-free Affect Detection in Cognitive Tutor Algebra. In Proceedings of the 5th International Conference on Educational Data Mining, International Educational Data Mining Society, 126--133.
[9]
Joseph Bergin. 2000. Fourteen Pedagogical Patterns. In Proceedings of the 5th European Conference on Pattern Languages of Programs, Universitaetsverlag Konstanz, Germany, 1--49.
[10]
Joseph Bergin. 2003. Teaching polymorphism with elementary design patterns. In Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications (OOPSLA '03). ACM, New York, NY, 167--169.
[11]
Joseph Bergin, Helen Sharp, Jane Chandler, Marianna Sipos, Jutta Eckstein, Markus Völter, Mary Lynn Manns, Eugene Wallingford, and Klaus Marquardt. 2012. Pedagogical patterns: advice for educators. Joseph Bergin Software Tools.
[12]
Benjamin S. Bloom. 1974. Time and learning. American psychologist, 29(9), 682.
[13]
Jan Borchers. 2001. A Pattern Approach to Interaction Design. John Wiley & Sons, Inc., New York, NY.
[14]
Zachary Broderick, Christine O'Connor, Courtney Mulcahy, Neil Heffernan, and Christina Heffernan. 2011. Increasing parent engagement in student learning using an intelligent tutoring system. Journal of Interactive Learning Research 22, 4, Association for the Advancement of Computing in Education, Chesapeake, VA, 523--550.
[15]
Carnegie Learning. n.d. Retrieved February 1, 2015 from http://www.carnegielearning.com/
[16]
Hao Cen, Kenneth Koedinger, and Brian Junker. 2007. Is Over Practice Necessary?-Improving Learning Efficiency with the Cognitive Tutor through Educational Data Mining. Frontiers in Artificial Intelligence and Applications, 158, 511.
[17]
Eric S. Chung, Jason I. Hong, James Lin, Madhu K. Prabaker, James A. Landay, and Alan L. Liu. 2004. Development and evaluation of emerging design patterns for ubiquitous computing. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques (DIS '04). ACM, New York, NY, 233--242.
[18]
Ruth C. Clark, and Richard E. Mayer. 2011. E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. John Wiley & Sons.
[19]
Doug Clow. 2014. Data wranglers: human interpreters to help close the feedback loop. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK '14). ACM, New York, NY, 49--53.
[20]
Coursera - Free Online Courses from Top Universities. n.d. Retrieved February 1, 2015 from https://www.coursera.org/
[21]
Andy Dearden, Janet Finlay, Liz Allgar, and Barbara McManus. 2002. Evaluating pattern languages in participatory design. In CHI'02 extended abstracts on Human factors in computing systems, ACM, 664--665.
[22]
David E. DeLano. 1998. Patterns mining. The Pattern Handbook, 87--96.
[23]
Andrea De Lucia, Vincenzo Deufemia, Carmine Gravino, and Michele Risi. 2010. Improving behavioral design pattern detection through model checking. In 2010 14th European Conference on Software Maintenance and Reengineering (CSMR), IEEE, 664--665.
[24]
Michael Derntl. 2004. The Person-Centered e-Learning pattern repository: Design for reuse and extensibility. In World Conference on Educational Multimedia, Hypermedia and Telecommunications, Association for the Advancement of Computing in Education, Lugano, Switzerland, 3856--3861.
[25]
Milton B. Dobrin and Carl H. Savit. 1960. Introduction to geophysical prospecting. McGraw-Hill, New York, NY.
[26]
A. B. Broughton Edge and T. H. Laby. 1931. The Principles and Practice of Geophysical Prospecting. Cambridge University Press, New York, NY.
[27]
Jing Dong, Yongtao Sun, and Yajing Zhao. 2008. Design pattern detection by template matching. In Proceedings of the 2008 ACM symposium on Applied computing, ACM, 765--769.
[28]
Francesca A. Fontana and Marco Zanoni, M. 2011. A tool for design pattern detection and software architecture reconstruction. Information sciences, 181(7), 1306--1324.
[29]
Sherri S. Frizell and Roland Hübscher. 2011. Using Design Patterns to Support E-Learning Design. In Instructional Design: Concepts, Methodologies, Tools and Applications. IGI-Global, Hershey, PA. 114--134.
[30]
Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1995. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Longman Publishing Co., Inc., Boston, MA.
[31]
Elissaveta Gourova, Kostadinka Toteva, and Yanka Todorova. 2012. Audit of knowledge flows and critical business processes. In Proceedings of the 17th European Conference on Pattern Languages of Programs. ACM, New York, NY.
[32]
Neil T. Heffernan and Cristina L. Heffernan. 2014. The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and Teaching. International Journal of Artificial Intelligence in Education 24, 4, 470--497.
[33]
Georgina Holden, Nicole Schadewitz, and Chrysi Rapanta. 2010. Patterns for the creation of elearning content and activities in a university setting. In Proceedings of the 15th European Conference on Pattern Languages of Programs (EuroPLoP '10). ACM, New York, NY, USA.
[34]
Takashi Iba and Mami Sakamoto. 2011. Learning patterns III: a pattern language for creative learning. In Proceedings of the 18th Conference on Pattern Languages of Programs (PLoP '11). ACM, New York, NY, USA.
[35]
Paul Salvador Inventado and Peter Scupelli. 2015. Towards an open, collaborative repository for online learning system design patterns. eLearning Papers, 42(Design Patterns for Open Online Teaching), 14--27.
[36]
Paul Salvador Inventado and Peter Scupelli. in press. Using Big Data to Produce Patterns for the ASSISTments Online Learning System. In Proceedings of the 22nd Conference on Pattern Languages of Programs (PLoP'15).
[37]
Norman L. Kerth and Ward Cunningham. 1997. Using patterns to improve our architectural vision. Software, IEEE, 14(1), 53--59.
[38]
Kenneth R. Koedinger, John R. Anderson, William H. Hadley, and Mary A. Mark. 1997. Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8, 30--43.
[39]
Kenneth R. Koedinger, Ryan S. J. D. Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper. 2010. A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining, CRC Press, Boca Raton, FL, 43--56.
[40]
Kenneth R. Koedinger, Albert T. Corbett, and Charles Perfetti. 2012. The Knowledgeo Learning Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science 36, 5, Blackwell Publishing Ltd., Hoboken, NJ, USA, 757--798.
[41]
Christian Kohls. 2013. The theories of design patterns and their practical implications exemplified for e-learning patterns. Ph.D. Dissertation. Catholic University of Eichstätt-Ingolstadt, Bavaria, Germany.
[42]
Christian Köppe and Mariëlle Nijsten. 2012. A pattern language for teaching in a foreign language: part 1. In Proceedings of the 17th European Conference on Pattern Languages of Programs (EuroPLoP '12). ACM, New York, NY.
[43]
Tibor Kunert. 2009. User-Centered Interaction Design Patterns for Interactive Digital Television Applications (1st ed.). Springer-Verlag, London, UK.
[44]
Niki Lambropoulos and Panayiotis Zaphiris. 2006. User-Centered Design of Online Learning Communities. IGI Publishing, Hershey, PA, USA.
[45]
Kathleen A. Larson, Frances P. Trees, and D. Scott Weaver. 2008. Continuous feedback pedagogical patterns. In Proceedings of the 15th Conference on Pattern Languages of Programs, ACM.
[46]
Shoujing Li, Xiaolu Xiong, and Joseph Beck. 2013. Modeling student retention in an environment with delayed testing. Educational Data Mining 2013. International Educational Data Mining Society, 328--329.
[47]
Michael J. Mahemoff and Lorraine J. Johnston. 1998. Principles for a Usability-Oriented Pattern Language. In Proceedings of the Australasian Conference on Computer Human Interaction (OZHCI '98). IEEE Computer Society, Washington, DC.
[48]
Michael Mendicino, Leena Razzaq, and Neil. T. Heffernan. 2009a. Improving learning from homework using intelligent tutoring systems. Journal of Research on Technology in Education, 331--346.
[49]
Michael Mendicino, Leena Razzaq, and Neil T. Heffernan. 2009b. A comparison of traditional homework to computer-supported homework. Journal of Research on Technology in Education (JRTE), 41, 3, 331--359.
[50]
Pat Morgan and Steven Ritter. 2002. An experimental study of the effects of Cognitive Tutor® Algebra I on student knowledge and attitude. Carnegie Learning, Inc., Pittsburgh, PA.
[51]
Theresa Neil. 2014. Mobile Design Pattern Gallery: UI Patterns for Smartphone Apps. O'Reilly Media, Inc., Sebastopol, CA.
[52]
Jaclyn Ocumpaugh, Ryan S. J. D. Baker, Sujith Gowda, Neil Heffernan and Cristina Heffernan. 2014. Population validity for Educational Data Mining models: A case study in affect detection. British Journal of Educational Technology 45, no. 3. Wiley-Blackwell, Hoboken, NJ, 487--501.
[53]
Brian C. Patrick, Ellen A. Skinner, and James P. Connell. 1993. What motivates children's behavior and emotion? Joint effects of perceived control and autonomy in the academic domain. Journal of Personality and social Psychology 65, 4, 781--791.
[54]
Reinhard Pekrun, Thomas Goetz, Wolfram Titz, and Raymond P. Perry. 2002. Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational psychologist 37, 2 (2002), Taylor & Francis, Philadelphia, PA, 91--105.
[55]
Rosalind W. Picard. 1997. Affective computing. MIT press.
[56]
Rahul Ramachandran, John Rushing, Amy Lin, Helen Conover, Xiang Li, Sara Graves, U.S. Nair, Kwo-Sen Kuo, and Deborah K. Smith. 2013. Data Prospecting--A Step Towards Data Intensive Science.Selected Topics in Applied Earth Observations and Remote Sensing, IEEE Journal of, 6(3), 1233--1241.
[57]
Doug Rohrer and Kelli Taylor. 2006. The effects of over-learning and distributed practice on the retention of mathematics knowledge. Applied Cognitive Psychology, 20, 1209--1224.
[58]
Lawrence C. Rowan, Simon J. Hook, Michael J. Abrams, and John C. Mars. 2003. Mapping hydrothermally altered rocks at Cuprite, Nevada, using the advanced spaceborne thermal emission and reflection radiometer (ASTER), a new satellite-imaging system. Economic Geology, 98, 5, 1019--1027.
[59]
Hank Sarkis. 2004. Cognitive Tutor Algebra 1 Program Evaluation: Miami-Dade County Public Schools. The Reliability Group, Lighthouse Point, FL.
[60]
Mike Sharkey. 2011. Academic analytics landscape at the University of Phoenix. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK '11). ACM, New York, NY, 122--126.
[61]
Colin Shearer. 2000. The CRISP-DM model: the new blueprint for data mining. Journal of data warehousing, 5(4), 13--22.
[62]
John A. Sloboda, Jane W. Davidson, Michael J. A. Howe, and Derek G. Moore. 1996. The role of practice in the development of performing musicians. British journal of psychology, 87(2), 287--310.
[63]
Sweller, J., & Cooper, G. A. 1985. The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59--89.
[64]
John Sweller. 2004. Instructional design consequences of an analogy between evolution by natural selection and human cognitive architecture. Instructional science, 32(1-2), 9--31.
[65]
Jenifer Tidwell. 2010. Designing interfaces. O'Reilly Media, Inc., Sebastopol, CA.
[66]
Michael Tuffiash, Roy W. Roring, and K. Anders Ericsson. 2007. Expert performance in SCRABBLE: implications for the study of the structure and acquisition of complex skills. Journal of Experimental Psychology: Applied, 13(3), 124.
[67]
Douglas K. Van Duyne, James A. Landay, and Jason I. Hong. 2007. The Design of Sites. Prentice Hall, Upper Saddle River, NJ.
[68]
Martijn Van Welie, Gerrit C. Van Der Veer, and Anton Eliëns. 2001. Patterns as tools for user interface design. In Tools for Working with Guidelines. Springer-Verlag, London, UK, 313--324.
[69]
Kalyan Veeramachaneni, Franck Dernoncourt, Colin Taylor, Zachary Pardos, and Una-May O'Reilly. 2013. Moocdb: Developing data standards for mooc data science. In Workshops Proceedings of AIED 2013, 17--24.
[70]
Skyler Whorton. 2013. Can a computer adaptive assessment system determine, better than traditional methods, whether students know mathematics skills? Master's thesis, Computer Science Department, Worcester Polytechnic Institute.
[71]
Wen-Hsiung Wu, Yen-Chun Jim Wu, Chun-Yu Chen, Hao-Yun Kao, Che-Hung Lin, and Sih-Han Huang. 2012. Review of trends from mobile learning studies: A meta-analysis.Computers & Education, 59(2), 817--827.
[72]
Michael V. Yudelson, Kenneth R. Koedinger, and Geoffrey J. Gordon. 2013. Individualized bayesian knowledge tracing models. In Artificial Intelligence in Education, Springer Berlin Heidelberg, 171--180.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EuroPLoP '15: Proceedings of the 20th European Conference on Pattern Languages of Programs
July 2015
714 pages
ISBN:9781450338479
DOI:10.1145/2855321
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. ASSISTments
  2. online learning systems
  3. pattern evaluation
  4. pattern mining
  5. pattern prospecting
  6. pattern writing

Qualifiers

  • Research-article

Funding Sources

Conference

EuroPLoP 2015

Acceptance Rates

Overall Acceptance Rate 216 of 354 submissions, 61%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)16
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media