research-article

Public Access

Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments

Authors:

David H. Smith, IV,

Craig ZillesAuthors Info & Claims

ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1

Pages 417 - 423

https://doi.org/10.1145/3587102.3588819

Published: 30 June 2023 Publication History

Abstract

In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is common for these problems to include incorrect blocks of code, typically referred to as "distractors," alongside the correct blocks. However, the utility of these distractors and their impact on a problems difficulty has yet to be thoroughly investigated. This study contributes to filling this gap by comparing performance, time spent, and item discrimination statistics for 32 pairs of Parsons problems from CS1 Python exams and quizzes. Our findings indicate that the inclusion of distractors has a large impact on the amount of time students spend on the questions and a low to moderate impact on score. Additionally, problems without distractors were already found to have high discrimination and including distractors did little to improve their discrimination. These findings suggest that the inclusion of distractors does little to improve the quality of these problems as exam questions but may have a negative impact on students by causing them to spend significantly more time on the problems and reducing the time they have for the rest of the exam.

References

[1]

Runestone Academy. [n.d.]. Python for everybody - interactive. https://runestone.academy/ns/books/published/py4e-int/index.html

[2]

Mary J Allen and Wendy M Yen. 2001. Introduction to measurement theory. Waveland Press.

[3]

Robert L Brennan. 1972. A generalized upper-lower item discrimination index. Educational and Psychological Measurement, Vol. 32, 2 (1972), 289--303.

[4]

Derek C Briggs, Alicia C Alonzo, Cheryl Schwab, and Mark Wilson. 2006. Diagnostic assessment with ordered multiple-choice items. Educational Assessment, Vol. 11, 1 (2006), 33--63.

[5]

Richard F Burton. 2001. Do item-discrimination indices really help us to improve our tests? Assessment & Evaluation in Higher Education, Vol. 26, 3 (2001), 213--220.

[6]

Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika, Vol. 16, 3 (1951), 297--334.

[7]

Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the fourth international workshop on computing education research. 113--124.

Digital Library

[8]

Yuemeng Du, Andrew Luxton-Reilly, and Paul Denny. 2020. A review of research on parsons problems. In Proceedings of the Twenty-Second Australasian Computing Education Conference. 195--202.

Digital Library

[9]

Robert L Ebel. 1967. The relation of item discrimination to test reliability. Journal of educational measurement, Vol. 4, 3 (1967), 125--128.

[10]

Robert L Ebel and David A Frisbie. 1972. Essentials of educational measurement. Prentice-Hall Englewood Cliffs, NJ.

[11]

Dirk Eddelbuettel and Alton Barbehenn. 2020. An R Autograder for PrairieLearn. arXiv preprint arXiv:2003.06500 (2020).

[12]

Max D Engelhart. 1965. A COMPARISON OF SEVERAL ITEM DISCRIMINATION INDICES 1. Journal of Educational Measurement, Vol. 2, 1 (1965), 69--76.

[13]

Barbara J Ericson, Lauren E Margulieux, and Jochen Rick. 2017. Solving parsons problems versus fixing and writing code. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research. 20--29.

Digital Library

[14]

Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017. Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, Vol. 87, 6 (2017), 1082--1116.

[15]

Bert F Green, Carolyn R Crone, and Valerie Greaud Folk. 1989. A method for studying differential distractor functioning. Journal of Educational Measurement, Vol. 26, 2 (1989), 147--160.

[16]

Louis Guttman and Izchak M Schlesinger. 1967. Systematic construction of distractors for ability and achievement test items. Educational and Psychological Measurement, Vol. 27, 3 (1967), 569--580.

[17]

Kyle James Harms, Jason Chen, and Caitlin L Kelleher. 2016. Distractors in Parsons problems decrease learning efficiency for young novice programmers. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 241--250.

Digital Library

[18]

Angelica Hotiu. 2006. The relationship between item difficulty and discrimination indices in multiple-choice tests in a physical science course. Ph.D. Dissertation. Citeseer.

[19]

Diana Kornbrot. 2014. Point biserial correlation. Wiley StatsRef: Statistics Reference Online (2014).

[20]

Raymond Lister, Tony Clear, Dennis J Bouvier, Paul Carter, Anna Eckerdal, Jana Jacková, Mike Lopez, Robert McCartney, Phil Robbins, Otto Sepp"al"a, et al. 2010. Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer. ACM SIGCSE Bulletin, Vol. 41, 4 (2010), 156--173.

Digital Library

[21]

Nelson Lojo and Armando Fox. 2022. Teaching Test-Writing as a Variably-Scaffolded Programming Pattern. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1. 498--504.

Digital Library

[22]

Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the fourth international workshop on computing education research. 101--112.

Digital Library

[23]

Frederic M Lord. 1953. An application of confidence intervals and of maximum likelihood to the estimation of an examinee's ability. Psychometrika, Vol. 18, 1 (1953), 57--76.

[24]

Frederic M Lord and Melvin R Novick. 2008. Statistical theories of mental test scores. IAP. 330--332 pages.

[25]

Wajiha Mahjabeen, Saeed Alam, Usman Hassan, Tahira Zafar, Rubab Butt, Sadaf Konain, and Myedah Rizvi. 2017. Difficulty index, discrimination index and distractor efficiency in multiple choice questions. Annals of PIMS-Shaheed Zulfiqar Ali Bhutto Medical University, Vol. 13, 4 (2017), 310--315.

[26]

Geofferey N Masters. 1988. Item discrimination: When more is worse. Journal of Educational Measurement, Vol. 25, 1 (1988), 15--29.

[27]

Dale Parsons and Patricia Haden. 2006. Parson's programming puzzles: a fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 157--163.

Digital Library

[28]

Seth Poulsen, Shubhang Kulkarni, Geoffrey Herman, and Matthew West. 2023. Efficient Feedback and Partial Credit Grading of Proof Blocks Problems. (2023).

[29]

Adi Setiawan. 2014. Simulation study of item validity testing and item discrimination index. In 1st International Conference on Electrical Engineering, Computer Science and Informatics 2014. Institute of Advanced Engineering and Science.

[30]

David H Smith IV and Craig Zilles. 2023. Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 924--930.

[31]

Marie Tarrant, James Ware, and Ahmed M Mohammed. 2009. An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC medical education, Vol. 9, 1 (2009), 1--8.

[32]

Rashmi Vyas and Avinash Supe. 2008. Multiple choice questions: a literature review on the optimal number of options. Natl Med J India, Vol. 21, 3 (2008), 130--3.

[33]

Nathaniel Weinman, Armando Fox, and Marti Hearst. 2020. Exploring challenging variations of parsons problems. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 1349--1349.

Digital Library

[34]

Nathaniel Weinman, Armando Fox, and Marti A Hearst. 2021. Improving instruction of programming patterns with faded parsons problems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--4.

Digital Library

[35]

Matthew West, Geoffrey L Herman, and Craig Zilles. 2015. Prairielearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition. 26--1238.

[36]

Craig B Zilles, Matthew West, Geoffrey L Herman, and Timothy Bretl. 2019. Every University Should Have a Computer-Based Testing Facility. In CSEDU (1). 414--420.

Cited By

Wu ZSmith DMonga MLonati VBarendsen ESheard JPaterson J(2024)Evaluating Micro Parsons Problems as Exam QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653583(674-680)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653583
Wu ZEricson B(2024)SQL Puzzles: Evaluating Micro Parsons Problems With Different Feedbacks as Practice for NovicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641910(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641910
Ericson BPearce JRodger SCsizmadia AGarcia RGutierrez FLiaskos KPadiyath AScott MSmith DWarriem JZavaleta Bernuy ALaakso MMonga MSimon Sheard J(2023)Multi-Institutional Multi-National Studies of Parsons ProblemsProceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3623762.3633498(57-107)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1145/3623762.3633498

Index Terms

Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1
SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

In this paper, we make three contributions related to the selection and use of distractors (lines of code reflecting common errors or misconceptions) in Parsons problems. First, we demonstrate a process by which templates for creating distractors can be ...
Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers
ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

Parsons problems are an increasingly popular method for helping inexperienced programmers improve their programming skills. In Parsons problems, learners are given a set of programming statements that they must assemble into the correct order. Parsons ...
Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments
CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 1

Parsons problems are a commonly used problem type typically used in introductory computer science courses. They involve organizing blocks containing segments of code to form a program. These questions often use ''distractors'' which are plausible, but ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1

June 2023

694 pages

ISBN:9798400701382

DOI:10.1145/3587102

General Chairs:
Mikko-Jussi Laakso
University of Turku, Finland
,
Mattia Monga
University of Milan, Italy
,
Program Chairs:
Simon
Unaffiliated, Australia
,
Judithe Sheard
Monash University, Australia

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 June 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ITiCSE 2023

Sponsor:

SIGCSE

ITiCSE 2023: Innovation and Technology in Computer Science Education

July 7 - 12, 2023

Turku, Finland

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
133
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)5

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu ZSmith DMonga MLonati VBarendsen ESheard JPaterson J(2024)Evaluating Micro Parsons Problems as Exam QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653583(674-680)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653583
Wu ZEricson B(2024)SQL Puzzles: Evaluating Micro Parsons Problems With Different Feedbacks as Practice for NovicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641910(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641910
Ericson BPearce JRodger SCsizmadia AGarcia RGutierrez FLiaskos KPadiyath AScott MSmith DWarriem JZavaleta Bernuy ALaakso MMonga MSimon Sheard J(2023)Multi-Institutional Multi-National Studies of Parsons ProblemsProceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3623762.3633498(57-107)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1145/3623762.3633498

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents