research-article

Superficial Code-guise: Investigating the Impact of Surface Feature Changes on Students' Programming Question Scores

Authors:

Craig ZillesAuthors Info & Claims

SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education

Pages 3 - 9

https://doi.org/10.1145/3408877.3432413

Published: 05 March 2021 Publication History

Abstract

Assessing student performance on programming questions is important for introductory computer science courses, both for student learning and for ensuring students demonstrate competence. Part of being a competent programmer includes the ability to transfer learning from solved to analogous problems. Additionally, particularly in computer-based and online assessment, mitigating cheating efforts is another important consideration. One way to mitigate cheating is by randomly selecting from large pools of equivalent questions. In order to produce large pools of questions quickly, we used a permutation strategy to rapidly make new question variants by altering existing questions' surface features. In this work, we present the results of our first set of surface feature permuted questions in an introductory Python course. We find surface feature permutations to be an effective way to produce questions of a similar difficulty to other new questions for students while mitigating potential cheating. However, we also see permutations expose potential student knowledge fragility and transfer concerns, as performance on permutations of homework questions is not strictly better than performance on questions that are entirely new on assessments

References

[1]

Miriam Bassok. 2003. Analogical Transfer in Problem Solving .Cambridge University Press, 343--370. https://doi.org/10.1017/CBO9780511615771.012

[2]

Jeffrey Bonar and Elliot Soloway. 1985. Preprogramming Knowledge: A Major Source of Misconceptions in Novice Programmers. Human--Computer Interaction, Vol. 1, 2 (1985), 133--161. https://doi.org/10.1207/s15327051hci0102_3

Digital Library

[3]

Guy Brousseau and Michael Otte. 1991. The Fragility of Knowledge . In Mathematical Knowledge: Its Growth Through Teaching, Alan J. Bishop, Stieg Mellin-Olsen, and Joop Van Dormolen (Eds.). Springer Netherlands, Dordrecht, 11--36. https://doi.org/10.1007/978--94-017--2195-0_2

[4]

T. Busjahn, R. Bednarik, A. Begel, M. Crosby, J. H. Paterson, C. Schulte, B. Sharif, and S. Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. In 2015 IEEE 23rd International Conference on Program Comprehension. 255--265.

Digital Library

[5]

Binglin Chen, Sushmita Azad, Max Fowler, Matthew West, and Craig Zilles. 2020. Learning to Cheat: Quantifying Changes in Score Advantage of Unproctored Assessments Over Time. In Proceedings of the Seventh ACM Conference on Learning @ Scale (Virtual Event, USA) (L@S '20). Association for Computing Machinery, New York, NY, USA, 197--206. https://doi.org/10.1145/3386527.3405925

Digital Library

[6]

Binglin Chen, Matthew West, and Craig Zilles. 2018. How Much Randomization is Needed to Deter Collaborative Cheating on Asynchronous Exams?. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S '18). Association for Computing Machinery, New York, NY, USA, Article 62, bibinfonumpages10 pages. https://doi.org/10.1145/3231644.3231664

Digital Library

[7]

Michelene T. H. Chi, Paul J. Feltovich, and Robert Glaser. 1981. Categorization and Representation of Physics Problems by Experts and Novices*. Cognitive Science, Vol. 5, 2 (1981), 121--152. https://doi.org/10.1207/s15516709cog0502_2

[8]

Aparna Chirumamilla, Guttorm Sindre, and Anh Nguyen-Duc. 2020. Cheating in e-exams and paper exams: the perceptions of engineering students and teachers in Norway . Assessment & Evaluation in Higher Education, Vol. 0, 0 (Jan. 2020), 1--18. https://doi.org/10.1080/02602938.2020.1719975

[9]

Michael J. Clancy and Marcia C. Linn. 1999. Patterns and Pedagogy. In The Proceedings of the Thirtieth SIGCSE Technical Symposium on Computer Science Education (New Orleans, Louisiana, USA) (SIGCSE '99). ACM, New York, NY, USA, 37--42. https://doi.org/10.1145/299649.299673

Digital Library

[10]

Henry Corrigan-Gibbs, Nakull Gupta, Curtis Northcutt, Edward Cutrell, and William Thies. 2015. Deterring Cheating in Online Environments . https://doi.org/10.1145/2810239

[11]

Donald R. Cressey. 1950. The Criminal Violation of Financial Trust. American Sociological Review, Vol. 15, 6 (1950), 738--743.

[12]

E. Enström. 2013. Dynamic programming - Structure, difficulties and teaching. In 2013 IEEE Frontiers in Education Conference (FIE). 1857--1863. https://doi.org/10.1109/FIE.2013.6685158

[13]

Lena Feinman. 2018. Alternative to Proctoring in Introductory Statistics Community College Courses . Ph.D. Dissertation. Walden University.

[14]

Dedre Gentner. 1983. Structure-Mapping: A Theoretical Framework for Analogy*. Cognitive Science, Vol. 7, 2 (1983), 155--170. https://doi.org/10.1207/s15516709cog0702_3

[15]

B. Haberman and O. Muller. 2008. Teaching abstraction to novices: Pattern-based and ADT-based problem-solving processes. In 2008 38th Annual Frontiers in Education Conference. F1C--7--F1C--12. https://doi.org/10.1109/FIE.2008.4720415

[16]

M. K. Hartwig and J. Dunlosky. 2012. Study strategies of college students: Are self-testing and scheduling related to achievement? Psychonomic Bulletin and Review, Vol. 19 (2012), 126--134.

[17]

Eder HernÃ¡ndez, Esmeralda Campos, Pablo Barniol, and Genaro Zavala. 2020. The effect of similar surface features on studentsâ?? understanding of the interaction of charges with electric and magnetic fields. (2020). http://hdl.handle.net/11285/636305

[18]

Keith J. Holyoak and Kyunghee Koh. 1987. Surface and structural similarity in analogical transfer. Memory & Cognition, Vol. 15, 4 (July 1987), 332--340. https://doi.org/10.3758/BF03197035

[19]

Michelle Ichinco and Caitlin Kelleher. 2017. Towards better code snippets: Exploring how code snippet recall differs with programming experience. In Visual Languages and Human-Centric Computing (VL/HCC), 2017 IEEE Symposium on. IEEE, 37--41.

[20]

Matthew Inglis and Lara Alcock. 2012. Expert and Novice Approaches to Reading Mathematical Proofs . Journal for Research in Mathematics Education, Vol. 43, 4 (2012), 358--390. https://doi.org/10.5951/jresematheduc.43.4.0358

[21]

Nicole Johnson?Glauch, Dong San Choi, and Geoffrey Herman. 2020. How engineering students use domain knowledge when problem-solving using different visual representations. Journal of Engineering Education, Vol. 109, 3 (2020), 443--469. https://doi.org/10.1002/jee.20348

[22]

Cazembe Kennedy and Eileen T. Kraemer. 2018. What Are They Thinking? Eliciting Student Reasoning About Troublesome Concepts in Introductory Computer Science. In Proceedings of the 18th Koli Calling International Conference on Computing Education Research (Koli, Finland) (Koli Calling '18). Association for Computing Machinery, New York, NY, USA, Article 7, bibinfonumpages10 pages. https://doi.org/10.1145/3279720.3279728

[23]

Mark M Lanier. 2006. Academic integrity and distance learning. Journal of criminal justice education, Vol. 17, 2 (2006), 244--261.

[24]

J.T. Laverty, S.M. Underwood, R.L. Matz, L.A. Posey, J.H. Carmel, M.D. Caballero, C. L. Fata-Hartley, D. Ebert-May, S. E. Jardeleza, and M. M. Cooper. 2016. Characterizing college science assessments: The three-dimensional learning assessment protocol. PLoS ONE, Vol. 11, 9 (2016), e0162333. https://doi.org/10.1371/journal.pone.0162333

[25]

Chang J. Lee. 2018. Automated Randomization of Test Problems for Cheating Prevention . World Journal of Research and Review, Vol. 6, 2 (2018).

[26]

Andrew Luxton-Reilly and Andrew Petersen. 2017. The Compound Nature of Novice Programming Assessments. In Proceedings of the Nineteenth Australasian Computing Education Conference (Geelong, VIC, Australia) (ACE '17). Association for Computing Machinery, New York, NY, USA, 26--35. https://doi.org/10.1145/3013499.3013500

Digital Library

[27]

Harvey Mellar, Roumiana Peytcheva-Forsyth, Serpil Kocdar, Abdulkadir Karadeniz, and Blagovesna Yovkova. 2018. Addressing cheating in e-assessment using student authentication and authorship checking systems: teachers? perspectives. International Journal for Educational Integrity, Vol. 14, 1 (Feb. 2018), 2. https://doi.org/10.1007/s40979-018-0025-x

[28]

Timothy B. Michael and Melissa A. Williams. 2013. Student Equity: Discouraging Cheating in Online Courses . Administrative Issues Journal: Education, Practice, and Research, Vol. 3, 2 (2013). https://eric.ed.gov/?id=EJ1057085

[29]

Orna Muller. 2005. Pattern Oriented Instruction and the Enhancement of Analogical Reasoning. In Proceedings of the First International Workshop on Computing Education Research (Seattle, WA, USA) (ICER '05). Association for Computing Machinery, New York, NY, USA, 57--67. https://doi.org/10.1145/1089786.1089792

Digital Library

[30]

Nancy Pennington. 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive psychology, Vol. 19, 3 (1987), 295--341.

[31]

Yizhou Qian and James Lehman. 2017. Students? Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Trans. Comput. Educ., Vol. 18, 1, Article 1 (Oct. 2017), bibinfonumpages24 pages. https://doi.org/10.1145/3077618

Digital Library

[32]

Joacim Ramberg and Bitte Modin. 2019. School effectiveness and student cheating: Do students? grades and moral standards matter for this relationship? Social Psychology of Education, Vol. 22, 3 (July 2019), 517--538. https://doi.org/10.1007/s11218-019-09486--6

[33]

Anthony Robins, Janet Rountree, and Nathan Rountree. 2003. Learning and Teaching Programming: A Review and Discussion. Computer Science Education, Vol. 13, 2 (2003), 137--172. https://doi.org/10.1076/csed.13.2.137.14200 https://doi.org/10.1187/cbe.14-07-0110

[34]

Kshitij Sharma, Patrick Jermann, Marc-Antoine Nüssli, and Pierre Dillenbourg. 2012. Gaze Evidence for Different Activities in Program Understanding. In PPIG.

[35]

Andreas Stefik and Susanna Siebert. 2013. An Empirical Investigation into Programming Language Syntax. ACM Trans. Comput. Educ.13, 4, Article 19 (Nov. 2013), 40 pages. https://doi.org/10.1145/2534973

Digital Library

[36]

Jason M Stephens, Michael F Young, and Thomas Calabrese. 2007. Does moral judgment go offline when students are online? A comparative analysis of undergraduates' beliefs and behaviors related to conventional and digital cheating. Ethics & Behavior 17, 3 (2007), 233-254.

[37]

Ethel Tshukudu and Quintin Cutts. 2020. Understanding Conceptual Transfer for Students Learning New Programming Languages. In Proceedings of the 2020ACM Conference on International Computing Education Research(Virtual Event, New Zealand)(ICER '20). Association for Computing Machinery, New York, NY, USA, 227-237. https://doi.org/10.1145/3372782.3406270

Digital Library

[38]

Gary M. Velan, Philip Jones, H. Patrick McNeil, and Rakesh K. Kumar. 2008. Integrated online formative assessments in the biomedical sciences for medical students: benefits for learning. BMC Medical Education 8, 1 (Nov. 2008), 52. https://doi.org/10.1186/1472-6920-8-52

[39]

Chelley Vician, Debra D Charlesworth, and Paul Charlesworth. 2006. Students' perspectives of the influence of web-enhanced coursework on incidences of cheating. Journal of Chemical Education 83, 9 (2006), 1368. https://doi.org/10.1021/ed083p1368

[40]

George R Watson and James Sottile. 2010. Cheating in the digital age: Do students cheat more in online courses? Online Journal of Distance Learning Administration 13, 1 (2010).

[41]

Matthew West, Geoffrey L. Herman, and Craig Zilles. 2015. Prairie Learn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. ASEE Annual Conference and Exposition, Conference Proceedings 122nd ASEE Annual Conference and Exposition: Making Value for Society, 122nd ASEE Annual Conference and Exposition: Making Value for... (2015). 2015 122nd ASEE Annual Conference and Exposition; Conference date: 14-06-2015 Through 17-06-2015.

[42]

Michele Weston, Kevin C. Haudek, Luanna Prevost, Mark Urban-Lurain, and John Merrill. 2015. Examining the Impact of Question Surface Features on Students' Answers to Constructed-Response Questions on Photosynthesis. CBE Life Sciences Education 14, 2 (June 2015). https://doi.org/10.1187/cbe.14-07-0110

Cited By

Fowler MSmith DZilles CMonga MLonati VBarendsen ESheard JPaterson J(2024)Quickly Producing "Isomorphic" Exercises: Quantifying the Impact of Programming Question PermutationsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653617(178-184)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653617
Demirtaş MFowler MHu NCunningham K(2024)Validating, Refining, and Identifying Programming Plans Using Learning Curve Analysis on Code Writing DataProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671120(263-279)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671120
Karnalim OHandoyo EJohan MSurjawan D(2024)Plagiarism on Mobile Programming Assessments: Automated Detection2024 2nd International Conference on Technology Innovation and Its Applications (ICTIIA)10.1109/ICTIIA61827.2024.10761687(1-5)Online publication date: 12-Sep-2024
https://doi.org/10.1109/ICTIIA61827.2024.10761687
Show More Cited By

Index Terms

Superficial Code-guise: Investigating the Impact of Surface Feature Changes on Students' Programming Question Scores
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
        CS1
      2. Student assessment

Recommendations

Quickly Producing "Isomorphic" Exercises: Quantifying the Impact of Programming Question Permutations
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

Small, auto-gradable programming exercises provide a useful tool with which to assess students' programming skills in introductory computer science. To reduce the time needed to produce programming exercises of similar difficulty, previous research has ...
Impact of preprogramming course curriculum on learning in the first programming course

This paper reports the results of research that measured the value of integrating problem solving, algorithm development, pseudocode, and diagramming techniques into introductory computer science courses. The hypothesis was that the introduction of ...
Investigating Novice Programmers' Interaction with Programming Environments
ITiCSE '19: Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education

Learning computer programming can be challenging for novices. Students have to deal with theoretical aspects of programming and problem solving in general, as well as mastering the syntax of a programming language. However, the feedback students receive ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education

March 2021

1454 pages

ISBN:9781450380621

DOI:10.1145/3408877

General Chairs:
Mark Sherriff
University of Virginia, USA
,
Laurence D. Merkle
Air Force Institute of Technology, USA
,
Program Chairs:
Pamela Cutter
Kalamazoo College, USA
,
Alvaro Monge
California State University, Long Beach, USA
,
Judithe Sheard
Monash University, Australia

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 March 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article

Conference

SIGCSE '21

Sponsor:

SIGCSE

SIGCSE '21: The 52nd ACM Technical Symposium on Computer Science Education

March 13 - 20, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE TS 2025

Sponsor:
sigcse

The 56th ACM Technical Symposium on Computer Science Education

February 26 - March 1, 2025

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
400
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)5

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fowler MSmith DZilles CMonga MLonati VBarendsen ESheard JPaterson J(2024)Quickly Producing "Isomorphic" Exercises: Quantifying the Impact of Programming Question PermutationsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653617(178-184)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653617
Demirtaş MFowler MHu NCunningham K(2024)Validating, Refining, and Identifying Programming Plans Using Learning Curve Analysis on Code Writing DataProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671120(263-279)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671120
Karnalim OHandoyo EJohan MSurjawan D(2024)Plagiarism on Mobile Programming Assessments: Automated Detection2024 2nd International Conference on Technology Innovation and Its Applications (ICTIIA)10.1109/ICTIIA61827.2024.10761687(1-5)Online publication date: 12-Sep-2024
https://doi.org/10.1109/ICTIIA61827.2024.10761687
Li ZYu ZZhai LPeng XXu Z(2024)ICBench: Benchmarking Knowledge Mastery in Introductory Computer Science EducationBenchmarking, Measuring, and Optimizing10.1007/978-981-97-0316-6_1(1-17)Online publication date: 14-Feb-2024
https://doi.org/10.1007/978-981-97-0316-6_1
Karnalim O(2024)Identifying AI Generated Code with Parallel KNN Weight Outlier DetectionAdvanced Technologies and the University of the Future10.1007/978-3-031-71530-3_29(459-470)Online publication date: 17-Dec-2024
https://doi.org/10.1007/978-3-031-71530-3_29
Pereira FRodrigues LHenklain MFreitas HOliveira DCristea ACarvalho LIsotani SBenedict ADorodchi Mde Oliveira E(2023)Toward Human–AI Collaboration: A Recommender System to Support CS1 Instructors to Select Problems for Assignments and ExamsIEEE Transactions on Learning Technologies10.1109/TLT.2022.322412116:3_Part_2(457-472)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1109/TLT.2022.3224121
Hickman HMcKeown PBell T(2023)Beyond Question Shuffling: Randomization Techniques in Programming Assessment2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10342976(1-9)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10342976
Karnalim OSimon Chivers W(2023)Reporting less coincidental similarity to educate students about programming plagiarism and collusionComputer Science Education10.1080/08993408.2023.217806334:3(442-472)Online publication date: 21-Feb-2023
https://doi.org/10.1080/08993408.2023.2178063
Allen SKizilcec R(2023)A systemic model of academic (mis)conduct to curb cheating in higher educationHigher Education10.1007/s10734-023-01077-x87:5(1529-1549)Online publication date: 19-Jul-2023
https://doi.org/10.1007/s10734-023-01077-x
Karnalim OSimon Chivers WPanca B(2022)Educating Students about Programming Plagiarism and Collusion via Formative FeedbackACM Transactions on Computing Education10.1145/350671722:3(1-31)Online publication date: 9-Jun-2022
https://dl.acm.org/doi/10.1145/3506717
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents