research-article

Semi-supervised verified feedback generation

Authors:

Shalini Kaleeswaran,

Anirudh Santhiar,

Sumit GulwaniAuthors Info & Claims

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 739 - 750

https://doi.org/10.1145/2950290.2950363

Published: 01 November 2016 Publication History

Abstract

Students have enthusiastically taken to online programming lessons and contests. Unfortunately, they tend to struggle due to lack of personalized feedback. There is an urgent need of program analysis and repair techniques capable of handling both the scale and variations in student submissions, while ensuring quality of feedback.

Towards this goal, we present a novel methodology called semi-supervised verified feedback generation. We cluster submissions by solution strategy and ask the instructor to identify or add a correct submission in each cluster. We then verify every submission in a cluster against the instructor-validated submission in the same cluster. If faults are detected in the submission then feedback suggesting fixes to them is generated. Clustering reduces the burden on the instructor and also the variations that have to be handled during feedback generation. The verified feedback generation ensures that only correct feedback is generated.

We implemented a tool, named CoderAssist, based on this approach and evaluated it on dynamic programming assignments. We have designed a novel counter-example guided feedback generation algorithm capable of suggesting fixes to all faults in a submission. In an evaluation on 2226 submissions to 4 problems, CoderAssist could generate verified feedback for 1911 (85%) submissions in 1.6s each on an average. It does a good job of reducing the burden on the instructor. Only one submission had to be manually validated or added for every 16 submissions.

References

[1]

www.acm.org/public-policy/education-policy-committee.

[2]

A. Adam and J.-P. Laurent. LAURA, A system to debug student programs. Artificial Intelligence, 15(1-2):75 – 122, 1980.

Digital Library

[3]

A. V. Aho, M. S. Lam, R. Sethi, and J. D. Ullman. Compilers: Principles, Techniques, and Tools. Pearson, 2nd edition, 2006.

Digital Library

[4]

R. Alur, L. D’Antoni, S. Gulwani, D. Kini, and M. Viswanathan. Automated grading of DFA constructions. In IJCAI, pages 1976–1982. Springer, 2013.

Digital Library

[5]

A. Arcuri. On the automation of fixing software bugs. In ICSE Companion, pages 1003–1006. ACM, 2008.

Digital Library

[6]

E. T. Barr, M. Harman, Y. Jia, A. Marginean, and J. Petke. Automated software transplantation. In ISSTA, pages 257–269. ACM, 2015.

Digital Library

[7]

C. Barrett, Y. Fang, B. Goldberg, Y. Hu, A. Pnueli, and L. Zuck. TVOC: A translation validator for optimizing compilers. In CAV, pages 291–295. Springer-Verlag, 2005.

Digital Library

[8]

R. E. Bellman. Dynamic Programming. Dover Publications, Incorporated, 2003.

Digital Library

[9]

C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, pages 209–224. USENIX Association, 2008.

Digital Library

[10]

L. De Moura and N. Bjørner. Z3: An efficient SMT solver. In TACAS, pages 337–340. Springer, 2008.

Digital Library

[11]

V. Debroy and W. E. Wong. Using Mutation to Automatically Suggest Fixes for Faulty Programs. In ICST, pages 65–74. IEEE Computer Society, 2010.

Digital Library

[12]

E. L. Glassman, J. Scott, R. Singh, P. J. Guo, and R. C. Miller. OverCode: Visualizing variation in student solutions to programming problems at scale. ACM Trans. Comput.-Hum. Interact., 22(2):7:1–7:35, 2015.

Digital Library

[13]

E. L. Glassman, R. Singh, and R. C. Miller. Feature engineering for clustering student solutions. In Proceedings of the First ACM Conference on Learning @ Scale Conference, L@S ’14, pages 171–172. ACM, 2014.

Digital Library

[14]

D. Gopinath, Z. M. Malik, and S. Khurshid. Specification-based program repair using SAT. In TACAS, pages 173–188. Springer, 2011.

Digital Library

[15]

S. Gross, X. Zhu, B. Hammer, and N. Pinkwart. Cluster based feedback provision strategies in intelligent tutoring systems. In Intelligent Tutoring Systems, pages 699–700. Springer, 2012.

Digital Library

[16]

S. Gulwani and S. Juvekar. Bound analysis using backward symbolic execution. Technical Report MSR-TR-2009-156, October 2009.

[17]

S. Gulwani, I. Radiˇcek, and F. Zuleger. Feedback generation for performance problems in introductory programming assignments. In FSE, pages 41–51. ACM, 2014.

Digital Library

[18]

M. Harman, W. B. Langdon, and W. Weimer. Genetic programming for reverse engineering. In WCRE, pages 1–10. IEEE Computer Society, 2013.

[19]

H. He and N. Gupta. Automated debugging using path-based weakest preconditions. In FASE, pages 267–280. Springer, 2004.

[20]

J. Huang, C. Piech, A. Nguyen, and L. J. Guibas. Syntactic and functional variability of a million code submissions in a machine learning MOOC. In AIED. CEUR-WS.org, 2013.

[21]

P. Ihantola, T. Ahoniemi, V. Karavirta, and O. Seppälä. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93. ACM, 2010.

Digital Library

[22]

S. Jha, S. Gulwani, S. A. Seshia, and A. Tiwari. Oracle-guided Component-based Program Synthesis. In ICSE, pages 215–224, 2010.

Digital Library

[23]

B. Jobstmann, A. Griesmayer, and R. Bloem. Program Repair as a Game. In CAV, pages 287–294. Springer, 2005.

Digital Library

[24]

S. Kaleeswaran, V. Tulsian, A. Kanade, and A. Orso. MintHint: Automated synthesis of repair hints. In ICSE, pages 266–276. ACM, 2014.

Digital Library

[25]

R. Konighofer and R. Bloem. Automated error localization and correction for imperative programs. In FMCAD, pages 91–100. FMCAD Inc., 2011.

Digital Library

[26]

C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In CGO, pages 75–88. IEEE Computer Society, 2004.

Digital Library

[27]

C. Le Goues, T. N., S. Forrest, and W. Weimer. GenProg: A generic method for automatic software repair. IEEE Trans. Software Eng., pages 54 –72, 2012.

Digital Library

[28]

F. Logozzo and T. Ball. Modular and verified automatic program repair. In OOPSLA, pages 133–146. ACM, 2012.

Digital Library

[29]

F. Long and M. Rinard. Staged program repair with condition synthesis. In ESEC/FSE, pages 166–178. ACM, 2015.

Digital Library

[30]

F. Long and M. Rinard. Automatic patch generation by learning correct code. In POPL, pages 298–312. ACM, 2016.

Digital Library

[31]

S. Mechtaev, J. Yi, and A. Roychoudhury. DirectFix: Looking for simple program repairs. In ICSE, pages 448–458. IEEE Computer Society, 2015.

Digital Library

[32]

S. Mechtaev, J. Yi, and A. Roychoudhury. Angelix: Scalable multiline program patch synthesis via symbolic analysis. In ICSE, pages 691–701. ACM, 2016.

Digital Library

[33]

L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin. Convolutional neural networks over tree structures for programming language processing. In AAAI, pages 1287–1293. AAAI Press, 2016.

[34]

I. Narasamdya and A. Voronkov. Finding basic block and variable correspondence. In Static Analysis, pages 251–267. Springer, 2005.

Digital Library

[35]

G. C. Necula. Translation validation for an optimizing compiler. In PLDI, pages 83–94. ACM, 2000.

Digital Library

[36]

A. Nguyen, C. Piech, J. Huang, and L. Guibas. Codewebs: Scalable homework search for massive open online programming courses. In WWW, pages 491–502. ACM, 2014.

Digital Library

[37]

H. D. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra. SemFix: Program repair via semantic analysis. In ICSE, pages 772–781. IEEE Computer Society, 2013.

Digital Library

[38]

N. Partush and E. Yahav. Abstract semantic differencing via speculative correlation. In OOPSLA, pages 811–828. ACM, 2014.

Digital Library

[39]

Y. Pei, Y. Wei, C. A. Furia, M. Nordio, and B. Meyer. Code-based automated program fixing. In ASE, pages 392–395. IEEE Computer Society, 2011.

Digital Library

[40]

H. Peng, L. Mou, G. Li, Y. Liu, L. Zhang, and Z. Jin. Building program vector representations for deep learning. In Knowledge Science, Engineering and Management, pages 547–553. Springer, 2015.

Digital Library

[41]

C. Piech, J. Huang, A. Nguyen, M. Phulsuksombati, M. Sahami, and L. J. Guibas. Learning program embeddings to propagate feedback on student code. In ICML, pages 1093–1102. JMLR.org, 2015.

[42]

A. Pnueli, M. Siegel, and E. Singerman. Translation validation. In TACAS, pages 151–166. Springer-Verlag, 1998.

Digital Library

[43]

Y. Pu, R. Bodik, and S. Srivastava. Synthesis of first-order dynamic programming algorithms. In OOPSLA, pages 83–98. ACM, 2011.

Digital Library

[44]

K. Rivers and K. Koedinger. Automatic generation of programming feedback: A data-driven approach. In AIED, pages 4:50–4:59. CEUR-WS.org, 2013.

[45]

R. Samanta, O. Olivo, and E. Emerson. Cost-aware automatic program repair. In Static Analysis, Lecture Notes in Computer Science, pages 268–284. Springer International Publishing, 2014.

[46]

R. Singh, S. Gulwani, and A. Solar-Lezama. Automated feedback generation for introductory programming assignments. In PLDI, pages 15–26. ACM, 2013.

Digital Library

[47]

Y. Wei, Y. Pei, C. A. Furia, L. S. Silva, S. Buchholz, B. Meyer, and A. Zeller. Automated fixing of programs with contracts. In ISSTA, pages 61–72. ACM, 2010.

Digital Library

[48]

S. Xu and Y. S. Chee. Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Trans. Softw. Eng., 29(4):360–384, April 2003.

Digital Library

Cited By

Xie LLi CPei YZhang TPan MChristakis MPradel M(2024)BRAFAR: Bidirectional Refactoring, Alignment, Fault Localization, and Repair for Programming AssignmentsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680326(856-868)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680326
Kushwaha AJadon SKamal PSaini MShrimal V(2024)Comments and Feedback Verification System using Large Language Model2024 8th International Conference on Computational System and Information Technology for Sustainable Solutions (CSITSS)10.1109/CSITSS64042.2024.10816848(1-6)Online publication date: 7-Nov-2024
https://doi.org/10.1109/CSITSS64042.2024.10816848
Fan ZTan SRoychoudhury AJust RFraser G(2023)Concept-Based Automated Grading of CS-1 Programming AssignmentsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598049(199-210)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598049
Show More Cited By

Index Terms

Semi-supervised verified feedback generation

Recommendations

Verified systems by composition from verified components
ESEC/FSE-11: Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering

This paper presents an approach to integration of model checking into component-based development of software systems. This approach assists in development of highly reliable component-based software systems and reduces the complexity of verifying these ...
Verified systems by composition from verified components

This paper presents an approach to integration of model checking into component-based development of software systems. This approach assists in development of highly reliable component-based software systems and reduces the complexity of verifying these ...
Developing verified programs with Dafny
HILT '12: Proceedings of the 2012 ACM conference on High integrity language technology

Reasoning about programs is a fundamental skill that every software engineer needs. This tutorial provides participants an opportunity to get hands-on experience with Dafny, a tool that can help develop this skill.

Dafny is a programming language and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2016

1156 pages

ISBN:9781450342186

DOI:10.1145/2950290

General Chair:
Thomas Zimmermann
Microsoft Research, USA
,
Program Chairs:
Jane Cleland-Huang
University of Notre Dame, USA
,
Zhendong Su
University of California at Davis, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13 - 18, 2016

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
650
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)5

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie LLi CPei YZhang TPan MChristakis MPradel M(2024)BRAFAR: Bidirectional Refactoring, Alignment, Fault Localization, and Repair for Programming AssignmentsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680326(856-868)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680326
Kushwaha AJadon SKamal PSaini MShrimal V(2024)Comments and Feedback Verification System using Large Language Model2024 8th International Conference on Computational System and Information Technology for Sustainable Solutions (CSITSS)10.1109/CSITSS64042.2024.10816848(1-6)Online publication date: 7-Nov-2024
https://doi.org/10.1109/CSITSS64042.2024.10816848
Fan ZTan SRoychoudhury AJust RFraser G(2023)Concept-Based Automated Grading of CS-1 Programming AssignmentsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598049(199-210)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598049
Milovančević DKunčak V(2023)Proving and Disproving Equivalence of Functional Programming AssignmentsProceedings of the ACM on Programming Languages10.1145/35912587:PLDI(928-951)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591258
Zhang AChen YOney S(2023)VizProg: Identifying Misunderstandings By Visualizing Students’ Coding ProgressProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581516(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581516
Li LLiu HLi KJiang YSun R(2023)Generating Concise Patches for Newly Released Programming AssignmentsIEEE Transactions on Software Engineering10.1109/TSE.2022.315352249:1(450-467)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3153522
Heo JJeong HChoi DLee E(2023)REFERENT: Transformer-based Feedback Generation using Assignment Information for Programming Course2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET)10.1109/ICSE-SEET58685.2023.00035(101-106)Online publication date: May-2023
https://doi.org/10.1109/ICSE-SEET58685.2023.00035
Bahrehvar MMoshirpour M(2023)Agile Teaching: Automated Student Support and Feedback Generation2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10343212(1-9)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10343212
Taniguchi YMinematsu TOkubo FShimada A(2022)Visualizing Source-Code Evolution for Understanding Class-Wide Programming ProcessesSustainability10.3390/su1413808414:13(8084)Online publication date: 1-Jul-2022
https://doi.org/10.3390/su14138084
Zhang JLi DKolesar JShi HPiskac R(2022)Automated Feedback Generation for Competition-Level CodeProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3560425(1-13)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3560425
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents