research-article

Benchmarking Introductory Programming Exams: How and Why

Authors:

Peter Klemperer,

Martijn Stegeman,

Daniel ZingaroAuthors Info & Claims

ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

Pages 154 - 159

https://doi.org/10.1145/2899415.2899473

Published: 11 July 2016 Publication History

Abstract

Ten selected questions have been included in 13 introductory programming exams at seven institutions in five countries. The students' results on these questions, and on the exams as a whole, lead to the development of a benchmark against which the exams in other introductory programming courses can be assessed. We illustrate some potential benefits of comparing exam performance against this benchmark, and show other uses to which it can be put, for example to assess the size and the overall difficulty of an exam. We invite others to apply the benchmark to their own courses and to share the results with us.

References

[1]

Bennedsen, J. and Caspersen, M.E. (2007). Failure rates in introductory programming. SIGCSE Bulletin, 39:2, 32--36.

Digital Library

[2]

Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, E., Sanders, K., Seppälä, O., Simon, B. and Thomas, L. (2004). A multi-national study of reading and tracing skills in novice programmers. SIGCSE Bulletin, 36:4, 119--150.

Digital Library

[3]

Lister, R., Corney, M., Curran, J., D'Souza, D., Fidge, C., Gluga, R., Hamilton, M., Harland, J., Hogan, J., Kay, J., Murphy, T., Roggenkamp, M., Sheard, J., Simon and Teague, D. (2012). Toward a shared understanding of competency in programming: An invitation to the BABELnot project. 14th Australasian Computing Education Conference (ACE 2012), 53--60.

Digital Library

[4]

McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Ben-David Kolikant, Y., Laxer, C., Thomas, L., Utting, I. and Wilusz, T. (2001). A multi-national, multi-institutional study assessment of programming skills of first-year CS students. SIGCSE Bulletin - Working Group reports: Making inroads to improve computing education, 33:4, 125--140.

Digital Library

[5]

Oliver, R. and Towers, S. (2000). Benchmarking ICT literacy in tertiary learning settings. 17th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education (ASCILITE 2000), 381--390.

[6]

Petersen, A., Craig, M. and Zingaro, D. (2011). Reviewing CS1 exam question content. 42nd ACM Technical Symposium on Computer Science Education (SIGCSE'11), 631--636.

Digital Library

[7]

Sheard, J., Simon, Carbone, A., Chinn, D., Clear, T., Corney, M., D'Souza, D., Fenwick, J., Harland, J., Laakso, M.-J. and Teague, D. (2013). How difficult are exams? A framework for assessing the complexity of introductory programming exams. 15th Australasian Computing Education Conference (ACE 2013), 145--154.

Digital Library

[8]

Sheard, J., Simon, Dermoudy, J., D'Souza, D., Hu, M., and Parsons, D. (2014). Benchmarking a set of exam questions for introductory programming. 16th Australasian Computing Education Conference (ACE 2014), 113--121.

Digital Library

[9]

Sim, S.E., Easterbrook, S. and Holt, R. (2003). Using benchmarking to advance research: a challenge to software engineering. 25th International Conference on Software Engineering, 74--83.

Digital Library

[10]

Simon, Sheard, J., Carbone, A., D'Souza, D., Harland, J. and Laakso, M.-J. (2012). Can computing academics assess the difficulty of programming examination questions? 11th Koli Calling International Conference on Computing Education Research, 160--163.

Digital Library

[11]

Simon, Sheard, J., D'Souza, D., Lopez, M., Luxton-Reilly, A., Putro I.H., Robbins, P., Teague, D., and Whalley, J. (2015). How (not) to write an introductory programming exam. 17th Australasian Computing Education Conference (ACE 2015), 137--146.

[12]

Simon and Snowdon, S. (2011). Explaining program code: giving students the answer helps -- but only just. Seventh International Computing Education Research Workshop (ICER 2011), 93--99.

Digital Library

[13]

Simon, B., Clancy, M., McCartney, R., Morrison, B., Richards, B., and Sanders, K. (2010). Making sense of data structures exams. Sixth International Computing Education Research workshop (ICER 2010), 97--105.

Digital Library

[14]

Watson, C. and Li, F.W. (2014). Failure rates in introductory programming revisited. 19th ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '14), 39--44.

Digital Library

[15]

Whalley, J., Lister, R., Thompson, E., Clear, T., Robbins, P., Kumar, P.K.A. and Prasad, C. (2006). An Australasian study of reading and comprehension skills in novice programmers, using the Bloom and SOLO taxonomies. Eighth Australasian Computing Education conference (ACE 2006), 243--251.

Digital Library

Cited By

Simon Sheard JLuxton-Reilly ASzabo C(2023)Computing Education Research in AustralasiaPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_17(373-394)Online publication date: 5-Jan-2023
https://doi.org/10.1007/978-3-031-25336-2_17
Mühling ASchulte CBennedsen JBudde LGroße-Bölting GIhantola PFalkner N(2019)Assessing students' understanding of object structuresProceedings of the 19th Koli Calling International Conference on Computing Education Research10.1145/3364510.3364511(1-10)Online publication date: 21-Nov-2019
https://dl.acm.org/doi/10.1145/3364510.3364511
Jackova JKanalikova A(2019)Introductory Programming Exams and Their Benchmarking in Slovakia2019 17th International Conference on Emerging eLearning Technologies and Applications (ICETA)10.1109/ICETA48886.2019.9039988(279-284)Online publication date: Nov-2019
https://doi.org/10.1109/ICETA48886.2019.9039988
Show More Cited By

Index Terms

Benchmarking Introductory Programming Exams: How and Why
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Benchmarking Introductory Programming Exams: Some Preliminary Results
ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

The programming education literature includes many observations that pass rates are low in introductory programming courses, but few or no comparisons of student performance across courses. This paper addresses that shortcoming. Having included a small ...
Oracle8i Certified Professional DBA Practice Exams
Oracle8i Certified Professional DBA Practice Exams

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

July 2016

394 pages

ISBN:9781450342315

DOI:10.1145/2899415

General Chairs:
Alison Clear
Eastern Institute of Technology, New Zealand
,
Ernesto Cuadros-Vargas
San Pablo Catholic University, Peru
,
Program Chairs:
Janet Carter
University of Kent, UK
,
Yvan Tupac
San Pablo Catholic University, Peru

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Bilkent University: Bilkent University
SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ITiCSE '16

Sponsor:

Bilkent University
SIGCSE

ITiCSE '16: Innovation and Technology in Computer Science Education Conference 2016

July 11 - 13, 2016

Arequipa, Peru

Acceptance Rates

ITiCSE '16 Paper Acceptance Rate 56 of 147 submissions, 38%;

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Upcoming Conference

ITiCSE '25

Sponsor:
sigcse

Innovation and Technology in Computer Science Education

June 27 - July 2, 2025

Nijmegen , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
307
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)2

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Simon Sheard JLuxton-Reilly ASzabo C(2023)Computing Education Research in AustralasiaPast, Present and Future of Computing Education Research10.1007/978-3-031-25336-2_17(373-394)Online publication date: 5-Jan-2023
https://doi.org/10.1007/978-3-031-25336-2_17
Mühling ASchulte CBennedsen JBudde LGroße-Bölting GIhantola PFalkner N(2019)Assessing students' understanding of object structuresProceedings of the 19th Koli Calling International Conference on Computing Education Research10.1145/3364510.3364511(1-10)Online publication date: 21-Nov-2019
https://dl.acm.org/doi/10.1145/3364510.3364511
Jackova JKanalikova A(2019)Introductory Programming Exams and Their Benchmarking in Slovakia2019 17th International Conference on Emerging eLearning Technologies and Applications (ICETA)10.1109/ICETA48886.2019.9039988(279-284)Online publication date: Nov-2019
https://doi.org/10.1109/ICETA48886.2019.9039988
Rubio M(2019)Automatic Categorization of Introductory Programming StudentsKinderchirurgie10.1007/978-3-030-20005-3_31(302-311)Online publication date: 28-Apr-2019
https://doi.org/10.1007/978-3-030-20005-3_31
Luxton-Reilly ASimon Albluwi IBecker BGiannakos MKumar AOtt LPaterson JScott MSheard JSzabo CRößling GScharlau B(2018)Introductory programming: a systematic literature reviewProceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education10.1145/3293881.3295779(55-106)Online publication date: 2-Jul-2018
https://dl.acm.org/doi/10.1145/3293881.3295779
Luxton-Reilly ABecker BCao YMcDermott RMirolo CMühling APetersen ASanders KSimon Whalley JSheard JKorhonen A(2018)Developing Assessments to Determine Mastery of Programming FundamentalsProceedings of the 2017 ITiCSE Conference on Working Group Reports10.1145/3174781.3174784(47-69)Online publication date: 30-Jan-2018
https://dl.acm.org/doi/10.1145/3174781.3174784
Simon Sheard JD'Souza DKlemperer PPorter LSorva JStegeman MZingaro DSheard JTenenberg JChinn DDorn B(2016)Benchmarking Introductory Programming ExamsProceedings of the 2016 ACM Conference on International Computing Education Research10.1145/2960310.2960337(103-111)Online publication date: 25-Aug-2016
https://dl.acm.org/doi/10.1145/2960310.2960337

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten