skip to main content
10.1145/3576050.3576063acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article

How Do Teachers Use Dashboards Enhanced with Data Storytelling Elements According to their Data Visualisation Literacy Skills?

Published: 13 March 2023 Publication History

Abstract

There is a proliferation of learning analytics (LA) dashboards aimed at supporting teachers. Yet, teachers still find it challenging to make sense of LA dashboards, thereby making informed decisions. Two main strategies to address this are emerging: i) upskilling teachers’ data literacy; ii) improving the explanatory design features of current dashboards (e.g., adding visual cues or text) to minimise the skills required by teachers to effectively use dashboards. While each approach has its own trade-offs, no previous work has explored the interplay between the dashboard design and such "data skills". In this paper, we explore how teachers with varying visualisation literacy (VL) skills use LA dashboards enhanced with (explanatory) data storytelling elements. We conducted a quasi-experimental study with 23 teachers of varied VL inspecting two versions of an authentic multichannel dashboard enhanced with data storytelling elements. We used an eye-tracking device while teachers inspected the students’ data captured from Zoom and Google Docs, followed by interviews. Results suggest that high VL teachers adopted complex exploratory strategies and were more sensitive to subtle inconsistencies in the design; while low VL teachers benefited the most from more explicit data storytelling guidance such as accompanying complex graphs with narrative and semantic colour encoding.

References

[1]
J. Ahn, H. Nguyen, 2021. From visible to understandable: Designing for teacher agency in education data visualizations. Contemporary Issues in Technology and Teacher Education 21, 1 (2021), 155–186.
[2]
H. Bao, Y. Li, 2021. The effects of a learning analytics dashboard on teachers’ diagnosis and intervention in computer-supported collaborative learning. Technology, Pedagogy and Education 30, 2 (2021), 287–303. https://doi.org/10.1080/1475939x.2021.1902383
[3]
O. Barral, S. LallÉ, 2021. Effect of Adaptive Guidance and Visualization Literacy on Gaze Attentive Behaviors and Sequential Patterns on Magazine-Style Narrative Visualizations. ACM Transactions on Interactive Intelligent Systems 11, 3-4 (Dec. 2021), 1–46. https://doi.org/10.1145/3447992
[4]
T. Bartindale, P. Chen, 2021. ZoomSense: A Scalable Infrastructure for Augmenting Zoom. In Proceedings of the 29th ACM International Conference on Multimedia (Virtual Event, China) (MM ’21). Association for Computing Machinery, New York, NY, USA, 3771–3774. https://doi.org/10.1145/3474085.3478332
[5]
R. Bender and S. Lange. 2001. Adjusting for multiple testing—when and how?Journal of clinical epidemiology 54, 4 (2001), 343–349.
[6]
V. Braun and V. Clarke. 2012. Thematic Analysis. In APA Handbook of Research Methods in Psychology, Vol. 2., H. Cooper, P. Camic, D. Long, A. Panter, D. Rindskopf, and K. Sher (Eds.). American Psychological Association, Washington, USA, 57–71. https://doi.org/10.1037/13620-004
[7]
S. Buckingham Shum, R. Ferguson, 2019. Human-Centred Learning Analytics. Journal of Learning Analytics 6, 2 (Jul. 2019), 1–9. https://doi.org/10.18608/jla.2019.62.1
[8]
F. Campos, J. Ahn, 2021. Making Sense of Sensemaking: Understanding How K–12 Teachers and Coaches React to Visual Analytics. Journal of Learning Analytics (July 2021), 1–21. https://doi.org/10.18608/jla.2021.7113
[9]
D. Ceneda, T. Gschwandtner, 2017. Characterizing Guidance in Visual Analytics. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 111–120. https://doi.org/10.1109/TVCG.2016.2598468
[10]
Q. Chen, Z. Li, 2019. Designing narrative slideshows for learning analytics. In 2019 IEEE Pacific Visualization Symposium (PacificVis). IEEE, 237–246. https://doi.org/10.1109/pacificvis.2019.00036
[11]
A. Choudhry, M. Sharma, 2021. Once Upon A Time In Visualization: Understanding the Use of Textual Narratives for Causality. IEEE Transactions on Visualization and Computer Graphics 27, 2 (feb 2021), 1332–1342. https://doi.org/10.1109/tvcg.2020.3030358
[12]
Y. Dimitriadis, R. Martínez-Maldonado, 2021. Human-Centered Design Principles for Actionable Learning Analytics. In Research on E-Learning and ICT in Education, Thrasyvoulos Tsiatsos, Stavros Demetriadis, Anastasios Mikropoulos, and Vasileios Dagdilelis (Eds.). Springer International Publishing, Cham, 277–296. https://doi.org/10.1007/978-3-030-64363-8_15
[13]
D. Donohoe and E. Costello. 2020. Data Visualisation Literacy in Higher Education: An Exploratory Study of Understanding of a Learning Dashboard Tool. International Journal of Emerging Technologies in Learning (iJET) 15, 17 (Sept. 2020), 115. https://doi.org/10.3991/ijet.v15i17.15041
[14]
R. A. Dourado, R. L. Rodrigues, 2021. A teacher-facing learning analytics dashboard for process-oriented feedback in online learning. In LAK21: 11th International Learning Analytics and Knowledge Conference. 482–489. https://doi.org/10.1145/3448139.3448187
[15]
V. Echeverria, R. Martinez-Maldonado, 2018. Exploratory versus Explanatory Visual Learning Analytics: Driving Teachers’ Attention through Educational Data Storytelling. Journal of Learning Analytics 5, 3 (Nov. 2018). https://doi.org/10.18608/jla.2018.53.6
[16]
V. Echeverria, R. Martinez-Maldonado, 2018. Driving Data Storytelling from Learning Design. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge. 131–140. https://doi.org/10.1145/3170358.3170380
[17]
R. J. Feise. 2002. Do multiple outcome measures require p-value adjustment?BMC medical research methodology 2, 1 (2002), 1–4.
[18]
G. M. Fernandez Nieto, K. Kitto, 2022. Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling. In LAK22: 12th International Learning Analytics and Knowledge Conference. ACM, Online USA, 219–229. https://doi.org/10.1145/3506860.3506895
[19]
E. E. Firat, A. Joshi, 2022. Interactive Visualization Literacy: The State-of-the-Art. Information Visualization 21, 3 (July 2022), 285–310. https://doi.org/10.1177/14738716221081831
[20]
M. J. Gomez, J. A. Ruipérez-Valiente, 2021. Applying learning analytics to detect sequences of actions and common errors in a geometry game. Sensors 21, 4 (2021), 1025. https://doi.org/10.3390/s21041025
[21]
C. Herodotou, B. Rienties, 2019. A Large-Scale Implementation of Predictive Learning Analytics in Higher Education: The Teachers’ Role and Perspective. Educational Technology Research and Development 67, 5 (Oct. 2019), 1273–1306. https://doi.org/10.1007/s11423-019-09685-0
[22]
R. Kaliisa, A. I. Mørch, 2022. ‘My Point of Departure for Analytics Is Extreme Skepticism’: Implications Derived from An Investigation of University Teachers’ Learning Analytics Perspectives and Design Practices. Technology, Knowledge and Learning 27, 2 (June 2022), 505–527. https://doi.org/10.1007/s10758-020-09488-w
[23]
S. Kardan, S. Lallé, 2021. EMDAT: Eye Movement Data Analysis Toolkit (1.x). Zenodo. https://doi.org/10.5281/zenodo.4699774
[24]
H. Khosravi, S. B. Shum, 2022. Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence 3 (2022), 100074. https://doi.org/10.1016/j.caeai.2022.100074
[25]
C. A. Knoop-van Campen, A. Wise, 2021. The equalizing effect of teacher dashboards on feedback in K-12 classrooms. Interactive Learning Environments(2021), 1–17. https://doi.org/10.1080/10494820.2021.1931346
[26]
S. Lalle, D. Toker, 2021. Gaze-Driven Adaptive Interventions for Magazine-Style Narrative Visualizations. IEEE Transactions on Visualization and Computer Graphics 27, 6 (June 2021), 2941–2952. https://doi.org/10.1109/TVCG.2019.2958540
[27]
S. Lee, S.-H. Kim, 2017. VLAT: Development of a Visualization Literacy Assessment Test. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 551–560. https://doi.org/10.1109/TVCG.2016.2598920
[28]
M. Liu, S. Han, 2021. The Current Landscape of Research and Practice on Visualizations and Dashboards for Learning Analytics. In Visualizations and Dashboards for Learning Analytics. Springer, 23–46. https://doi.org/10.1007/978-3-030-81222-5_2
[29]
E. B. Mandinach and L. M. Abrams. [n. d.]. Data Literacy and Learning Analytics. In The Handbook of Learning Analytics(2 ed.), Charles Lang, George Siemens, Alyssa Friend Wise, Dragan GaÅ¡ević, and Agathe Merceron (Eds.). SoLAR, 196–204. https://www.solaresearch.org/publications/hla-22/hla22- chapter19/ Section: 19.
[30]
R. Martinez-Maldonado, V. Echeverria, 2020. From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–15. https://doi.org/10.1145/3313831.3376148
[31]
N. McDonald, S. Schoenebeck, 2019. Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–23. https://doi.org/10.1145/3359174
[32]
I. Molenaar and C. A. N. Knoop-van Campen. 2019. How Teachers Make Dashboard Information Actionable. IEEE Transactions on Learning Technologies 12, 3 (July 2019), 347–355. https://doi.org/10.1109/TLT.2018.2851585
[33]
I. G. Ndukwe and B. K. Daniel. 2020. Teaching Analytics, Value and Tools for Teacher Data Literacy: A Systematic and Tripartite Approach. International Journal of Educational Technology in Higher Education 17, 1(2020), 1–31. https://doi.org/10.1186/s41239-020-00201-6
[34]
A. Ojo and B. Heravi. 2018. Patterns in Award Winning Data Storytelling: Story Types, Enabling Tools and Competences. Digital Journalism 6, 6 (July 2018), 693–718. https://doi.org/10.1080/21670811.2017.1403291
[35]
A. Olsen. 2012. The Tobii I-VT fixation filter. Tobii Technology 21(2012), 4–19.
[36]
S. Pozdniakov, R. Martinez-Maldonado, 2021. Question-driven learning analytics: Designing a teacher dashboard for online breakout rooms. In 2021 International Conference on Advanced Learning Technologies (ICALT). IEEE, 176–178.
[37]
S. Pozdniakov, R. Martinez-Maldonado, 2022. The Question-driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers’ Inquiry?. In LAK22: 12th International Learning Analytics and Knowledge Conference. 175–185. https://doi.org/10.1145/3506860.3506885
[38]
S. Pozdniakov, R. Martinez-Maldonado, 2022. The Question-Driven Dashboard: How Can We Design Analytics Interfaces Aligned to Teachers’ Inquiry?. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). Association for Computing Machinery, New York, NY, USA, 175–185. https://doi.org/10.1145/3506860.3506885
[39]
M. J. Rodríguez-Triana, L. P. Prieto, 2018. The Teacher in the Loop: Customizing Multimodal Learning Analytics for Blended Learning. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge - LAK ’18. ACM Press, Sydney, New South Wales, Australia, 417–426. https://doi.org/10.1145/3170358.3170364
[40]
S. Z. Salas-Pilco, K. Xiao, 2022. Artificial Intelligence and Learning Analytics in Teacher Education: A Systematic Review. Education Sciences 12, 8 (2022), 569. https://doi.org/10.3390/educsci12080569
[41]
J. Saldaña. 2015. The Coding Manual for Qualitative Researchers. SAGE. https://doi.org/10.46743/2160-3715/2009.2856
[42]
H.-J. Schulz, M. Streit, 2013. Towards a characterization of guidance in visualization. In Poster at IEEE Conference on Information Visualization (InfoVis).
[43]
S. Sergis and D. G. Sampson. 2017. Teaching and Learning Analytics to Support Teacher Inquiry: A Systematic Literature Review. Learning analytics: Fundaments, applications, and trends (2017), 25–63. https://doi.org/10.1007/978-3-319-52977-6
[44]
E. B. d. Sousa, B. Alexandre, 2021. Applications of learning analytics in high schools: a Systematic Literature review. Frontiers in Artificial Intelligence 4 (2021), 737891. https://doi.org/10.3389/frai.2021.737891
[45]
A. Stefanile. 2020. The Transition from Classroom to Zoom and How It Has Changed Education. Journal of Social Science Research 16 (July 2020), 33–40. https://doi.org/10.24297/jssr.v16i.8789
[46]
Y.-S. Tsai and D. Gasevic. 2017. Learning Analytics in Higher Education — Challenges and Policies: A Review of Eight Learning Analytics Policies. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference. ACM, Vancouver British Columbia Canada, 233–242. https://doi.org/10.1145/3027385.3027400
[47]
A. van Leeuwen, C. A. Knoop-van Campen, 2021. How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K-12. Journal of Learning Analytics 8, 2 (2021), 6–21. https://doi.org/10.18608/jla.2021.7325
[48]
A. van Leeuwen and N. Rummel. 2019. Orchestration Tools to Support the Teacher during Student Collaboration: A Review. Unterrichtswissenschaft 47, 2 (April 2019), 143–158. https://doi.org/10.1007/s42010-019-00052-9
[49]
A. Van Leeuwen and N. Rummel. 2020. Comparing teachers’ use of mirroring and advising dashboards. In Proceedings of the tenth international conference on learning analytics & knowledge. 26–34. https://doi.org/10.1145/3375462.3375471
[50]
R. Voithofer and A. M. Golan. 2018. Data Sources for Educators: Mining Meaningful Data for Course and Program Decision Making. In Responsible Analytics and Data Mining in Education. Routledge, 83–100. https://doi.org/10.4324/9780203728703-7
[51]
K. Williamson and R. Kizilcec. 2022. A review of learning analytics dashboard research in higher education: Implications for justice, equity, diversity, and inclusion. In LAK22: 12th International Learning Analytics and Knowledge Conference. 260–270. https://doi.org/10.1145/3506860.3506900
[52]
A. F. Wise and Y. Jung. 2019. Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics 6, 2 (2019), 53–69. https://doi.org/10.18608/jla.2019.62.4
[53]
A. Wolff, J. Moore, 2016. Data literacy for learning analytics. In Proceedings of the sixth international conference on learning analytics & knowledge. 500–501. https://doi.org/10.1145/2883851.2883864

Cited By

View all
  • (2024)VisRecall++: Analysing and Predicting Visualisation Recallability from Gaze BehaviourProceedings of the ACM on Human-Computer Interaction10.1145/36556138:ETRA(1-18)Online publication date: 28-May-2024
  • (2024)Data Storytelling Editor: A Teacher-Centred Tool for Customising Learning Analytics Dashboard NarrativesProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636930(678-689)Online publication date: 18-Mar-2024
  • (2024)Harnessing Transparent Learning Analytics for Individualized Support through Auto-detection of Engagement in Face-to-Face Collaborative LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636894(392-403)Online publication date: 18-Mar-2024
  • Show More Cited By

Index Terms

  1. How Do Teachers Use Dashboards Enhanced with Data Storytelling Elements According to their Data Visualisation Literacy Skills?

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        LAK2023: LAK23: 13th International Learning Analytics and Knowledge Conference
        March 2023
        692 pages
        ISBN:9781450398657
        DOI:10.1145/3576050
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 13 March 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. dashboard
        2. data literacy
        3. data storytelling
        4. human-centred design
        5. learning analytics

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        LAK 2023

        Acceptance Rates

        Overall Acceptance Rate 236 of 782 submissions, 30%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)268
        • Downloads (Last 6 weeks)21
        Reflects downloads up to 27 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media