research-article

Reliability of self-rated experience and confidence as predictors for students’ performance in software engineering: Results from multiple controlled experiments on model comprehension with graduate and undergraduate students

Authors:

Jennifer Brings,

Patricia Aluko Obe,

Viktoria StenkovaAuthors Info & Claims

Empirical Software Engineering, Volume 26, Issue 4

https://doi.org/10.1007/s10664-021-09972-6

Published: 01 July 2021 Publication History

Abstract

Students’ experience is used in empirical software engineering research as well as in software engineering education to group students in either homogeneous or heterogeneous groups. To do so, students are commonly asked to self-rate their experience, as self-rated experience has been shown to be a good predictor for performance in programming tasks. Another experience-related measurement is participants’ confidence (i.e., how confident is the person that their given answer is correct). Hence, self-rated experience and confidence are used as selector or control variables throughout empirical software engineering research and software engineering education. In this paper, we analyze data from several student experiments conducted in the past years to investigate whether self-rated experience and confidence are also good predictors for students’ performance in model comprehension tasks. Our results show that while students can somewhat assess the correctness of a particular answer to one concrete question regarding a conceptual model (i.e., their confidence), their overall self-rated experience does not correlate with their actual performance. Hence, the use of the commonly used measurement of self-rated experience as a selector or control variable must be considered unreliable for model comprehension tasks.

References

[1]

Arisholm E, Gallis H, Dyba T, and Sjoberg DEvaluating pair programming with respect to system complexity and programmer expertiseIEEE Trans Softw Eng200733265-86https://doi.org/10.1109/TSE.2007.17

[2]

Bastarrica MC, Simmonds J (2019) Gender differences in self and peer assessment in a software engineering capstone course. In: 2019 IEEE/ACM 2nd international workshop on gender equality in software engineering (GE), pp 29–32.

[3]

Berander P (2004) Using students as subjects in requirements prioritization. In: Proceedings. 2004 international symposium on empirical software engineering, 2004. ISESE ’04, pp 167–176.

[4]

Bergersen GR and Gustafsson JEProgramming skill, knowledge, and working memory among professional software developers from an investment theory perspectiveJ Indiv Diff2011324201-209https://doi.org/10.1027/1614-0001/a000052

[5]

Bergersen GR, Hannay JE, Sjoberg D, Dyba T, Karahasanovic A (2011) Inferring skill from tests of programming performance: combining time and quality. In: 2011 international symposium on empirical software engineering and measurement, iSSN: 1938-6451, pp 305–314.

[6]

Bergersen GR, Sjøberg DIK, and Dybå TConstruction and validation of an instrument for measuring programming skillIEEE Trans Softw Eng201440121163-1184https://doi.org/10.1109/TSE.2014.2348997

[7]

Biffl SEvaluating defect estimation models with major defectsJ Syst Softw200365113-29https://doi.org/10.1016/S0164-1212(02)00025-0

[8]

Boud D and Falchikov N Quantitative studies of student self-assessment in higher education: a critical analysis of findings High Educ 1989 18 5 529-549

[9]

Bunse CUsing patterns for the refinement and translationof UML models: a controlled experimentEmpir Softw Eng2006112227-267https://doi.org/10.1007/s10664-006-6403-7

[10]

Byrne P and Lyons GThe effect of student attributes on success in programmingSIGCSE Bull200133349-52https://doi.org/10.1145/507758.377467

[11]

Campbell DT and Stanley JC Experimental and quasi-experimental designs for research 1963 Boston Houghton Mifflin

[12]

Carver J, Jaccheri L, Morasca S, Shull F (2003) Issues in using students in empirical studies in software engineering education. In: Software metrics symposium, 2003. Proceedings. Ninth international, pp 239–249.

[13]

Cen L, Ruta D, Powell L, Ng J (2015) Interaction driven composition of student groups for optimal groupwork learning performance. In: 2015 IEEE frontiers in education conference (FIE), pp 1–6.

[14]

Cook TD and Campbell DT Quasi-experimentation: design & analysis issues for field settings 1979 Boston Houghton Mifflin

[15]

Cushion CJ, Armour KM, and Jones RL Coach education and continuing professional development: Experience and learning to coach Quest 2003 55 3 215-230

[16]

Daun M, Brings J, Weyer T (2017) On the impact of the model-based representation of inconsistencies to manual reviews. In: Mayr H, Guizzardi G, Ma H, Pastor O (eds) Conceptual modeling. ER 2017. Lecture notes in computer science, vol 10650. Springer, Cham.

[17]

Daun M, Brings J, Krajinski L, Weyer T (2019a) On the benefits of using dedicated models in validation processes for behavioral specifications. In: Sutton Jr SM, Armbrust O, Hebig R (eds) Proceedings of the international conference on software and system processes, ICSSP 2019, Montreal, QC, Canada, May 25-26, 2019. IEEE / ACM, pp 44–53.

[18]

Daun M, Weyer T, and Pohl KImproving manual reviews in function-centered engineering of embedded systems using a dedicated review modelSoftw Syst Model20191863421-3459https://doi.org/10.1007/s10270-019-00723-2

[19]

Daun M, Brings J, Weyer T (2020) Do instance-level review diagrams support validation processes of cyber-physical system specifications: Results from a controlled experiment. In: Proceedings of the international conference on software and system processes, ICSSP 2020, Seoul, Republic of Korea, October 10-11, 2020. ACM, p 10.

[20]

DeVellis RF (2017) Scale development. 4th ed. Los Angeles: Sage

[21]

Dick M, Postema M, Miller J (2001) Improving student performance in software engineering practice. In: Proceedings 14th conference on software engineering education and training. ’In search of a software engineering profession’ (Cat. No.PR01059), pp 143–152.

[22]

Diedenhofen B and Musch Jcocor: A comprehensive solution for the statistical comparison of correlationsPLOS ONE20151041-12https://doi.org/10.1371/journal.pone.0121945

[23]

El Emam K and Madhavji NHAn instrument for measuring the success of the requirements engineering process in information systems developmentEmpir Softw Eng199613201-240https://doi.org/10.1007/BF00127446

[24]

Eskew RK and Faley RH Some determinants of student performance in the first college-Level financial accounting course Account Rev 1988 63 1 137-147

[25]

Falchikov N and Boud DStudent self-Assessment in higher education: a meta-analysisRev Educ Res1989594395-430https://doi.org/10.3102/00346543059004395

[26]

Feigenspan J, Kästner C, Liebig J, Apel S, Hanenberg S (2012) Measuring programming experience. In: 2012 20th IEEE international conference on program comprehension (ICPC), iSSN: 1092-8138, pp 73–82.

[27]

Feldt R, Zimmermann T, Bergersen GR, Falessi D, Jedlitschka A, Juristo N, Münch J, Oivo M, Runeson P, Shepperd M, Sjøberg DIK, and Turhan BFour commentaries on the use of students and professionals in empirical software engineering experimentsEmpir Softw Eng20182363801-3820https://doi.org/10.1007/s10664-018-9655-0

[28]

Field A (2013) Discovering statistics using IBM SPSS statistics. 4th ed. London: Sage

[29]

Fucci D, Turhan B, Oivo M (2015) On the effects of programming and testing skills on external quality and productivity in a test-driven development context. In: EASE ’15: proceedings of the 19th international conference on evaluation and assessment in software engineering. ACM.

[30]

Goodwin CJ, Goodwin KA (2016) Research in psychology methods and design. John Wiley & Sons, Hoboken, NJ, USA

[31]

Hagan D, Markham S (2000) Does it help to have some programming experience before beginning a computing degree program?. In: Proceedings of the 5th annual SIGCSE/SIGCUE conference on innovation and technology in computer science education (ITiCSE 2000). Association for Computing Machinery (ACM), pp 25–28

[32]

Hannay JE, Arisholm E, Engvik H, and Sjoberg DIEffects of personality on pair programmingIEEE Trans Softw Eng201036161-80https://doi.org/10.1109/TSE.2009.41

[33]

Höst M, Regnell B, and Wohlin C Using students as subjects-A comparative study of students and professionals in lead-Time impact assessment Empir Softw Eng 2000 5 3 201-214

[34]

ISO/IEC/IEEE (2010) International standard - Systems and software engineering – Vocabulary. In: ISO/IEC/IEEE 24765:2010(E),

[35]

ITU (2016) International telecommunication union recommendation z.120: Message Sequence Chart (MSC). Tech. Rep. Z120, International Telecommunication Union

[36]

James T, Galster M, Blincoe K, Miller G (2017) What is the perception of female and male software professionals on performance, team dynamics and job satisfaction? Insights from the trenches. In: 2017 IEEE/ACM 39th international conference on software engineering: software engineering in practice track (ICSE-SEIP), pp 13–22.

[37]

Jamieson S Likert scales: How to (ab) use them? Med Educ 2004 38 12 1217-1218

[38]

Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Shull F, Singer J, Sjøberg DIK (eds). Springer, London, pp 201–228

[39]

Jensen LP (2015) Using consultation in student groups to improve development of team work skills amongst more reluctant students. In: Proceedings of the 43rd SEFI annual conference 2015 - diversity in engineering education: An opportunity to face the new trends of engineering, SEFI 2015

[40]

Jørgensen M, Teigen KH, and Moløkken KBetter sure than safe? Over-confidence in judgement based software development effort prediction intervalsJ Syst Softw200470179-93https://doi.org/10.1016/S0164-1212(02)00160-7

[41]

Katira N, Williams L, Wiebe E, Miller C, Balik S, and Gehringer EOn understanding compatibility of student pair programmersSIGCSE Bull20043617-11https://doi.org/10.1145/1028174.971307

[42]

Kirschner PAEpistemology, practical work and Academic skills in science educationSci Educ199213273-299https://doi.org/10.1007/BF00430277

[43]

Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, and Rosenberg JPreliminary guidelines for empirical research in software engineeringIEEE Trans Softw Eng2002288721-734https://doi.org/10.1109/TSE.2002.1027796

[44]

Kumar AN (2008) The effect of using problem-solving software tutors on the self-confidence of female students. In: Proceedings of the 39th SIGCSE technical symposium on computer science education, SIGCSE ’08. Association for Computing Machinery, New York, pp 523–527.

[45]

Layman L, Williams L, Osborne J, Berenson S, Slaten K, Vouk M (2005) How and why collaborative software development impacts the software engineering course. In: Proceedings frontiers in education 35th annual conference, pp T4C–T4C.

[46]

Likert R (1932) A technique for the measurement of attitudes. Archives of psychology

[47]

Lumley T, Diehr P, Emerson S, and Chen LThe importance of the normality assumption in large public health data setsAnnu Rev Public Health2002231151-169https://doi.org/10.1146/annurev.publhealth.23.100901.140546

[48]

Marshall L, Pieterse V, Thompson L, and Venter MDExploration of participation in student software engineering teamsACM Trans Comput Educ2016162, Article 538https://doi.org/10.1145/2791396

[49]

Mcdowell C, Werner L, Bullock HE, Fernald J (2003) The impact of pair programming on student performance, perception and persistence. In: 25th international conference on software engineering, 2003. Proceedings, pp 602–607.

[50]

Mishra T, Kumar D, Gupta S (2014) Mining students’ data for prediction performance. In: 2014 Fourth international conference on advanced computing communication technologies, iSSN: 2327-0659, pp 255–262.

[51]

Mkpojiogu EOC and Hussain AAssessing students’ performance in software requirements engineering education using scoring rubricsAIP Conf Proc201718911020092https://doi.org/10.1063/1.5005425

[52]

Müller MMAre reviews an alternative to pair programming?Empir Softw Eng200494335-351https://doi.org/10.1023/B:EMSE.0000039883.47173.39

[53]

Morgan PJ and Cleave-Hogg DComparison between medical students’ experience, confidence and competenceMed Educ2002366534-539https://doi.org/10.1046/j.1365-2923.2002.01228.x

[54]

Newhall T, Meeden L, Danner A, Soni A, Ruiz F, Wicentowski R (2014) A support program for introductory cs courses that improves student performance and retains students from underrepresented groups. In: Proceedings of the 45th ACM technical symposium on computer science education, SIGCSE ’14. Association for Computing Machinery, New York, pp 433–438.

[55]

Nugroho ALevel of detail in UML models and its impact on model comprehension: a controlled experimentInf Softw Technol200951121670-1685https://doi.org/10.1016/j.infsof.2009.04.007

[56]

Osgood CE, Suci GJ, Tannenbaum PH (1957) The measurement of meaning. Chicago: University of Illinois Press

[57]

Pinto G, Ferreira C, Souza C, Steinmacher I, Meirelles P (2019) Training software engineers using open-source software: the students’ perspective. In: Proceedings of the 41st international conference on software engineering: software engineering education and training, ICSE-SEET ’19, event-place: Montreal, Quebec, Canada. IEEE Press, Piscataway, pp 147–157.

[58]

Polančič G, Heričko M, and Rozman IAn empirical examination of application frameworks success based on technology acceptance modelJ Syst Softw2010834574-584https://doi.org/10.1016/j.jss.2009.10.036

[59]

Raza M, Faria JP, Salazar R (2017) Helping software engineering students analyzing their performance data: tool support in an educational environment. In: Proceedings of the 39th international conference on software engineering companion, ICSE-C ’17, event-place: Buenos Aires, Argentina. IEEE Press, Piscataway, pp 241–243.

[60]

Rea LM Designing and conducting survey research: a comprehensive guide fourth edition 2014 4 San Francisco Jossey-Bass

[61]

Rex K and Roth RMThe relationship of computer experience and computer self-Efficacy to performance in introductory computer literacy coursesJ Res Comput Educ199831114-24https://doi.org/10.1080/08886504.1998.10782238

[62]

Ricca F, Penta MD, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by UML stereotypes. In: 29Th international conference on software engineering, 2007. ICSE 2007, pp 375–384.

[63]

Robson C (2016) Real world research. 4th ed. Chichester: Wiley

[64]

Runeson P (2003) Using students as experiment subjects – an analysis on graduate and freshmen student data. In: Proceedings 7th international conference on empirical assessment & evaluation in software engineering, pp 95–102

[65]

Salman I, Misirli AT, Juristo N (2015) Are students representatives of professionals in software engineering experiments?. In: 2015 IEEE/ACM 37Th IEEE international conference on software engineering (ICSE), vol 1, pp 666–676.

[66]

Sillito J, Murphy GC, and De Volder KAsking and answering questions during a programming change taskIEEE Trans Softw Eng2008344434-451https://doi.org/10.1109/TSE.2008.26

[67]

Sinha A and Smidts CAn experimental evaluation of a higher-ordered-typed-functional specification-based test-generation techniqueEmpir Softw Eng2006112173-202https://doi.org/10.1007/s10664-006-6401-9

[68]

Sjøberg DIK, Anda B, Arisholm E, Dybå T, Jorgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. In: Empirical software engineering, 2002. Proceedings. 2002. International Symposium n, pp 17–26.

[69]

Sjøberg DIK, Anda B, Arisholm E, Dybå T, Jørgensen M, Karahasanović A, Vokáč M (2003) Challenges and recommendations when increasing the realism of controlled software engineering experiments. In: Conradi R, Wang AI (eds) Empirical methods and studies in software engineering: experiences from ESERNET, Lecture Notes in Computer Science. Springer, Berlin, pp 24–38

[70]

Tichy WF Hints for reviewing empirical work in software engineering Empir Softw Eng 2000 5 4 309-312

[71]

Venkatesh V and Bala HTechnology acceptance model 3 and a research agenda on interventionsDecis Sci2008392273-315https://doi.org/10.1111/j.1540-5915.2008.00192.x

[72]

Webb NM, Nemer KM, Chizhik AW, and Sugrue BEquity issues in collaborative group assessment: group composition and performanceAm Educ Res J1998354607-651https://doi.org/10.3102/00028312035004607

[73]

Wieringa R (2010) Design science methodology: Principles and practice. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering, ICSE ’10, vol 2. ACM, New York, pp 493–494.

[74]

Witten IH, Frank E, Trigg LE, Hall MA, Holmes G, Cunningham SJ (1999) Weka: Practical machine learning tools and techniques with Java implementations. Working Paper, University of Waikato, Department of Computer Science, Hamilton, New Zealand, 99/11

[75]

Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, and Wesslén A Experimentation in software engineering: An introduction, Kluwer international series in software engineering, vol 6 2000 Boston Kluwer Academic

[76]

Zhang D, Fonseca P, Cuthbert L, Ketteridge S (2014) An investigation of the team knowledge and team performance of the Chinese engineering students in a senior technical module. In: 2014 IEEE frontiers in education conference (FIE) Proceedings., iSSN: 2377-634X, pp 1–8

Cited By

Metzger ALaufer JFeit FPohl K(2024)A User Study on Explainable Online Reinforcement Learning for Adaptive SystemsACM Transactions on Autonomous and Adaptive Systems10.1145/366600519:3(1-44)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3666005
Strickroth SMonga MLonati VBarendsen ESheard JPaterson J(2024)Exploring Students' Self-Confidence in Their Programming SolutionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653589(415-421)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653589
Raja JRaju ABrings JDaun M(2024)Extending Goal Models with Execution Orders: An Investigation of the Impact on ComprehensibilityAdvances in Conceptual Modeling10.1007/978-3-031-75599-6_17(219-228)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1007/978-3-031-75599-6_17
Show More Cited By

Recommendations

Exploring Students' Self-Confidence in Their Programming Solutions
ITiCSE 2024: Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1

Learning programming is perceived as hard by many students. To support students, many e-assessment and intelligent tutoring systems have been developed. These systems can automatically evaluate student submissions and provide feedback. Despite ...
Self-predicted and actual performance in an introductory programming course
ITiCSE '10: Proceedings of the fifteenth annual conference on Innovation and technology in computer science education

Students in a large introductory programming course were asked twice to predict their scores on the final exam: once at the beginning of a six-week module, and once at the end. In between, students in only one of the two lecture streams recorded ...
Enhancing the Clustering of Student Performance Using the Variation in Confidence
Intelligent Tutoring Systems
Abstract
While prior research has typically treated student self-confidence as a static measure, confidence is not identical in all situations. We study the degree to which confidence varies over time using entropy, investigating whether high variation in ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 26, Issue 4

Jul 2021

1061 pages

ISSN:1382-3256

Issue’s Table of Contents

© The Author(s) 2021.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2021

Accepted: 14 April 2021

Author Tags

Qualifiers

Research-article

Funding Sources

Universität Duisburg-Essen (3149)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Metzger ALaufer JFeit FPohl K(2024)A User Study on Explainable Online Reinforcement Learning for Adaptive SystemsACM Transactions on Autonomous and Adaptive Systems10.1145/366600519:3(1-44)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3666005
Strickroth SMonga MLonati VBarendsen ESheard JPaterson J(2024)Exploring Students' Self-Confidence in Their Programming SolutionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653589(415-421)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653589
Raja JRaju ABrings JDaun M(2024)Extending Goal Models with Execution Orders: An Investigation of the Impact on ComprehensibilityAdvances in Conceptual Modeling10.1007/978-3-031-75599-6_17(219-228)Online publication date: 29-Oct-2024
https://dl.acm.org/doi/10.1007/978-3-031-75599-6_17
Daun MBrings J(2023)Investigating Factors Influencing Students’ Assessment of Conceptual ModelsProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593960(470-474)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593960
Daun MBrings J(2023)Aggregating N-fold Requirements Inspection ResultsProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593465(339-347)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593465

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents