research-article

VG: Automatic Grading of D3 Visualizations<sc/><sc/>

Authors:

Vivian Pednekar,

Susanta Routray,

Mahdi Roozbahani,

Duen Horng ChauAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 30, Issue 1

Pages 617 - 627

https://doi.org/10.1109/TVCG.2023.3327181

Published: 26 October 2023 Publication History

Abstract

Manually grading D3 data visualizations is a challenging endeavor, and is especially difficult for large classes with hundreds of students. Grading an interactive visualization requires a combination of interactive, quantitative, and qualitative evaluation that are conventionally done manually and are difficult to scale up as the visualization complexity, data size, and number of students increase. We present V<sc>is</sc>G<sc>rader</sc>, a first-of-its kind automatic grading method for D3 visualizations that scalably and precisely evaluates the data bindings, visual encodings, interactions, and design specifications used in a visualization. Our method enhances students' learning experience, enabling them to submit their code frequently and receive rapid feedback to better inform iteration and improvement to their code and visualization design. We have successfully deployed our method and auto-graded D3 submissions from more than 4000 students in a visualization course at Georgia Tech, and received positive feedback for expanding its adoption.

References

[1]

CS444 - Data Visualization. University of Arizona. [Online]. Available: https://cscheid.net/courses/fall-2019/csc444/, 2019. Accessed on: Mar 02, 2023. 1.

[2]

CS171 - Visualization. Harvard University. [Online]. Available: https://www.cs171.org/2022/, 2022. Accessed on: March 2, 2023. 1.

[3]

CS5630/CS6630 - Visualization for Data Science. The University of Utah. [Online]. Available: https://www.dataviscourse.net/2022/syllabus/, 2022. Accessed on: Mar 02, 2023. 1.

[4]

CSE512 - Data Visualization. University of Washington. [Online]. Available: https://courses.cs.washington.edu/courses/cse512/22sp/, 2022. Accessed on: Mar 02, 2023. 1.

[5]

CS448B - Data Visualization. Stanford University. [Online]. Available: https://online.stanford.edu/courses/cs448b-data-visualization, 2023. Accessed on: Mar 02, 2023. 1.

[6]

CS6242 Data and Visual Analytics. Georgia Institute of Technology. [Online]. Available: https://poloclub.github.io/cse6242-2023spring-online/, 2023. Accessed on: Jan 08, 2023. 1, 2, 3.

[7]

EdDiscussion. [Online]. Available: https://edstem.org, 2023. Accessed on: Aug, 08, 2022. 7.

[8]

Enrollment history of CS6242 Data and Visual Analytics. Georgia Institute of Technology. [Online]. Available: https://poloclub.github.io/#cse6242, 2023. Accessed on: Jan 08, 2023. 3.

[9]

U. Z. Ahmed, N. Srivastava, R. Sindhgatta, and A. Karkare. Characterizing the pedagogical benefits of adaptive feedback for compilation errors by novice programmers. In Proc. ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET'20, pp. 139–150. ACM, New York, 2020. 2.

Digital Library

[10]

L. Battle, D. Feng, and K. Webber. Exploring d3 implementation challenges on stack overflow. In 2022 IEEE Visualization and Visual Analytics (VIS), pp. 1–5, 2022. 3.

[11]

Blackboard Inc. Blackboard LMS. [Online]. Available: https://www.blackboard.com/en-mea/teaching-learning/learning-management, 2023. Accessed on: Mar 01, 2023. 5.

[12]

M. Bostock, V. Ogievetsky, and J. Heer. D³ data-driven documents. IEEE Trans. Vis. Comput. Graph., 17 (12): pp. 2301–2309, 2011. 1, 3, 9.

Digital Library

[13]

Q. Chen, F. Sun, X. Xu, Z. Chen, J. Wang, and N. Cao. Vizlinter: A linter and fixer framework for data visualization. IEEE Trans. Vis. Comput. Graph., 28 (1): pp. 206–216, 2022. 3.

Digital Library

[14]

Z. Chen, Y. Wang, Q. Wang, Y. Wang, and H. Qu. Towards automated infographic design: Deep learning-based auto-extraction of extensible timeline. IEEE Trans. Vis. Comput. Graph., 26 (1): pp. 917–926, 2020. 9.

Digital Library

[15]

D2L Corporation. Brightspace LMS. [Online]. Available: https://www.d21.com/brightspace/, 2023. Accessed on: Mar 01, 2023. 5.

[16]

F. Elavsky. Chartability. [Online]. Available: https://chartability.github.io/POUR-CAF/, 2021. Accessed on: Jul 1, 2021. 9.

[17]

J. Gao, B. Pang, and S. S. Lumetta. Automated feedback framework for introductory programming courses. In Proc. ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE '16, pp. 53–58. ACM, New York, 2016. 2, 3.

Digital Library

[18]

P. Godefroid, N. Klarlund, and K. Sen. Dart: Directed automated random testing. SIGPLAN Not., 40 (6): pp. 213–223, 2005. 2.

Digital Library

[19]

Gradescope Inc. Gradescope. [Online]. Available: https://www.gradescope.com. 2020. Accessed on: May 01, 2020. 4, 6.

[20]

V. Gramoli, M. Charleston, B. Jeffries, I. Koprinska, M. McGrane, A. Radu, A. Viglas, and K. Yacef. Mining autograding data in computer science education. In Proc. Australasian Computer Science Week Multiconference, number 1 in ACSW '16, pp. 1–10. ACM, New York, 2016. 3, 7.

Digital Library

[21]

S. Gulwani, I. Radiček, and F. Zuleger. Feedback generation for performance problems in introductory programming assignments. In Proc. ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pp. 41–51. ACM, New York, 2014. 2.

Digital Library

[22]

J. Harper and M. Agrawala. Deconstructing and restyling d3 visualizations. in UIST '14, pp. 253–262. ACM, New York, 2014. 3, 5.

Digital Library

[23]

A. K. Hopkins, M. Correll, and A. Satyanarayan. “Visualint: Sketchy in situ annotations of chart construction errors”. In Computer Graphics Forum, vol. 39, pp. 219–228. Wiley Online Library, 2020. 3.

[24]

E. Hoque and M. Agrawala. Searching the visual style and structure of d3 visualizations. IEEE Trans. Vis. Comput. Graph., 26 (1): pp. 1236–1245, 2020. 3.

[25]

K. Hu, M. A. Bakker, S. Li, T. Kraska, and C. Hidalgo. Vizml: A machine learning approach to visualization recommendation. In Proc. CHI Conference on Human Factors in Computing Systems, CHI'19, pp. 1–12. ACM, New York, 2019. 9.

Digital Library

[26]

M. Hull, C. Guerin, J. Chen, S. Routray, and D. H. Chau. Towards automatic grading of d3.js visualizations. arXiv preprint arXiv:, 2021. 6.

[27]

Instructure Inc. Canvas LMS. [Online]. Available: https://www.instructure.com/canvas. 2023. Accessed on: Mar 01, 2023. 5.

[28]

D. Joyner. Teaching at Scale: Improving Access, Outcomes, and Impact Through Digital Instruction. Taylor & Francis, 2022. 8.

[29]

L. Y.-H. Lo, Y. Ming, and H. Qu. Learning vis tools: Teaching data visualization tutorials. In 2019 IEEE Visualization Conference (VIS), pp. 11–15, 2019. 1.

[30]

Y. Luo, X. Qin, N. Tang, and G. Li. Deepeye: Towards automatic data visualization. In 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 101–112, 2018. 9.

[31]

E. Maicus, M. Peveler, A. Aikens, and B. Cutler. Autograding interactive computer graphics applications. in SIGCSE '20, pp. 1145–1151. ACM, New York, 2020. 1, 3.

Digital Library

[32]

A. McNutt, G. Kindlmann, and M. Correll. Surfacing visualization mi-rages. in CHI '20. ACM, New York, 2020. 3.

Digital Library

[33]

H. Mei, Y. Ma, Y. Wei, and W. Chen. The design space of construction tools for information visualization: A survey. J. Vis. Lang. Comput., 44: pp. 120–132, 2018. 3.

[34]

Moodle PTY Ltd. Moodle LMS. [Online]. Available: https://moodle.com/solutions/lms/, 2023. Accessed on: Mar 01, 2023. 5.

[35]

D. Moritz. Draco: Formalizing Visualization Design Knowledge as Constraints. [Online]. Available: https://uwdata.github.io/draco/, 2018. Accessed on: Mar 09, 2023. 3.

[36]

D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe, and J. Heer. Formalizing visualization design knowledge as constraints: Actionable and extensible models in draco. IEEE Trans. Vis. Comput. Graph., 25 (1): pp. 438–448, 2019. 3.

Digital Library

[37]

Open JS Foundation. WebHint. [Online]. Available: https://https://webhint.io, 2023. Accessed on: Jun 29, 2023. 9.

[38]

S. Parihar, Z. Dadachanji, P. K. Singh, R. Das, A. Karkare, and A. Bhattacharya. Automatic grading and feedback using program repair for introductory programming courses. In Proc. ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE'17, pp. 92–97. ACM, New York, 2017. 2, 3.

Digital Library

[39]

A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer. Vega-lite: A grammar of interactive graphics. IEEE Trans. Vis. Comput. Graph., 23 (1): pp. 341–350, 2017. 3.

Digital Library

[40]

S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cortés. A survey on metamorphic testing. IEEE Trans. Softw. Eng., 42 (9): pp. 805–824, 2016. 3.

[41]

SGW Communications. Sakai LMS. [Online]. Available: https://www.sakailms.org, 2023. Accessed on: Mar 01, 2023. 5.

[42]

M. Sherman, S. Bassil, D. Lipman, N. Tuck, and F. Martin. Impact of auto-grading on an introductory computing course. J. Comput. Sci. Coll., 28 (6): pp. 69–75, 2013. 3,4,7.

Digital Library

[43]

S. Shin, S. Hong, and N. Elmqvist. Perceptual pat: A virtual human system for iterative visualization design. arXiv preprint arXiv:, 2023. 3, 9.

[44]

Software Freedom Conservancy. Selenium. [Online]. Available: https://www.selenium.dev, 2021. Accessed on: February 01, 2021. 1, 3.

[45]

Software Freedom Conservancy. Selenium Action Chain. [Online]. Available: https://www.selenium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.action_chains.html, 2021. Accessed on: March 14, 2021. 1,6.

[46]

T. Tang, R. Li, X. Wu, S. Liu, J. Knittel, S. Koch, T. Ertl, L. Yu, P. Ren, and Y. Wu. Plotthread: Creating expressive storyline visualizations using reinforcement learning. IEEE Trans. Vis. Comput. Graph., 27 (2): pp. 294–303, 2021. 9.

[47]

Q. Wang, Z. Chen, Y. Wang, and H. Qu. A survey on ml4vis: Applying machine learning advances to data visualization. IEEE Trans. Vis. Comput. Graph., 28 (12): pp. 5134–5153, 2022. 9.

Digital Library

[48]

C. Wilcox. The role of automation in undergraduate computer science education. in SIGCSE'15, pp. 90–95. ACM, New York, 2015. 2.

Digital Library

[49]

J. Wood, A. Kachkaev, and J. Dykes. Design exposition with literate visualization. IEEE Trans. Vis. Comput. Graph., 25 (1): pp. 759–768, 2019. 3.

Digital Library

Cited By

Wang ZMunechika DLee SChau D(2024)SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational NotebooksExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650848(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650848
McNutt AStone MHeer J(2024)Mixing Linters with GUIs: A Color Palette Design ProbeIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345631731:1(327-337)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456317

Index Terms

VG: Automatic Grading of D3 Visualizations

Index terms have been assigned to the content through auto-classification.

Recommendations

Deconstructing and restyling D3 visualizations
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

The D3 JavaScript library has become a ubiquitous tool for developing visualizations on the Web. Yet, once a D3 visualization is published online its visual style is difficult to change. We present a pair of tools for deconstructing and restyling ...
Visualizing Visualizations: User Interfaces for Managing and Exploring Scientific Visualization Data

The process of scientific visualization is inherently iterative. A good visualization comes from experimenting with visualization, rendering, and viewing parameters to bring out the most relevant information in the data. A good data visualization system ...
Visualizing the Scripts of Data Wrangling With Somnus
Data workers use various scripting languages for data transformation, such as SAS, R, and Python. However, understanding intricate code pieces requires advanced programming skills, which hinders data workers from grasping the idea of data transformation ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 30, Issue 1

Jan. 2024

1456 pages

Issue’s Table of Contents

1077-2626 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 26 October 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZMunechika DLee SChau D(2024)SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational NotebooksExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650848(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650848
McNutt AStone MHeer J(2024)Mixing Linters with GUIs: A Color Palette Design ProbeIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345631731:1(327-337)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456317

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents