article

Towards building a universal defect prediction model with rank transformed predictors

Authors:

Iman Keivanloo,

Ying ZouAuthors Info & Claims

Empirical Software Engineering, Volume 21, Issue 5

Pages 2107 - 2145

https://doi.org/10.1007/s10664-015-9396-2

Published: 01 October 2016 Publication History

Abstract

Software defects can lead to undesired results. Correcting defects costs 50 % to 75 % of the total software development budgets. To predict defective files, a prediction model must be built with predictors (e.g., software metrics) obtained from either a project itself (within-project) or from other projects (cross-project). A universal defect prediction model that is built from a large set of diverse projects would relieve the need to build and tailor prediction models for an individual project. A formidable obstacle to build a universal model is the variations in the distribution of predictors among projects of diverse contexts (e.g., size and programming language). Hence, we propose to cluster projects based on the similarity of the distribution of predictors, and derive the rank transformations using quantiles of predictors for a cluster. We fit the universal model on the transformed data of 1,385 open source projects hosted on SourceForge and GoogleCode. The universal model obtains prediction performance comparable to the within-project models, yields similar results when applied on five external projects (one Apache and four Eclipse projects), and performs similarly among projects with different context factors. At last, we investigate what predictors should be included in the universal model. We expect that this work could form a basis for future work on building a universal model and would lead to software support tools that incorporate it into a regular development workflow.

References

[1]

Akiyama F (1971) An example of software system debugging. In: Proceedings of the international federation of information processing societies congress, pp 353-359.

[2]

Alves T, Ypma C, Visser J (2010) Deriving metric thresholds from benchmark data. In: Proceedings of the 26th IEEE international conference on software maintenance, pp 1-10.

[3]

Arisholm E, Briand LC, Johannessen EB (2010) A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. J Syst Softw 83(1):2-17.

Digital Library

[4]

Baggen R, Correia J, Schill K, Visser J (2012) Standardized code quality benchmarking for improving software maintainability. Softw Qual J 20:287-307.

Digital Library

[5]

Bettenburg N, Hassan AE (2010) Studying the impact of social structures on software quality. In: Proceedings of the 18th IEEE international conference on program comprehension, ICPC '10, pp 124-133.

[6]

Bird C, Bachmann A, Aune E, Duffy J, Bernstein A, Filkov V, Devanbu P (2009) Fair and balanced?: bias in bug-fix datasets. In: Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, ESEC/FSE '09, pp 121-130.

[7]

Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494-509.

[8]

Cohen J (1988) Statistical power analysis for the behavioral sciences: Jacob Cohen, 2nd edn. Lawrence Erlbaum.

[9]

Cohen J (1992) A power primer. Psychol Bull 112(1):155-159.

[10]

Cruz A, Ochimizu K (2009) Towards logistic regression models for predicting fault-prone code across software projects. In: 3rd international symposium on empirical software engineering and measurement, 2009. ESEM 2009, pp 460-463.

[11]

D'Ambros M, Lanza M, Robbes R (2010) An extensive comparison of bug prediction approaches. In: Proceedings of the 7th IEEE working conference on mining software repositories, MSR'10, pp 31-41.

[12]

D'Ambros M, Lanza M, Robbes R (2012) Evaluating defect prediction approaches: a benchmark and an extensive comparison. Empir Softw Eng 17(4-5):531-577.

Digital Library

[13]

Denaro G, Pezzè M (2002) An empirical evaluation of fault-proneness models. In: Proceedings of the 24rd International Conference on Software Engineering, 2002. ICSE 2002, pp 241-251.

[14]

Hailpern B, Santhanam P (2002) Software debugging, testing, and verification. IBM Syst J 41(1):4-12.

Digital Library

[15]

Hall T, Beecham S, Bowes D, Gray D, Counsell S (2012) A systematic literature review on fault prediction performance in software engineering. IEEE Trans Softw Eng 38(6):1276-1304.

Digital Library

[16]

Hassan A (2009) Predicting faults using the complexity of code changes. In: Proceedings of the 31st IEEE international conference on software engineering, ICSE'09, pp 78-88.

[17]

He Z, Shu F, Yang Y, Li M, Wang Q (2012) An investigation on the feasibility of cross-project defect prediction. Autom Softw Eng 19(2):167-199.

Digital Library

[18]

He Z, Peters F, Menzies T, Yang Y (2013) Learning from open-source projects: an empirical study on defect prediction. In: 2013 ACM/IEEE international symposium on empirical software engineering and measurement, pp 45-54.

[19]

Herzig K, Just S, Zeller A (2013) It's not a bug, it's a feature: how misclassification impacts bug prediction. In: Proceedings of the 35th international conference on software engineering, ICSE '13, pp 392-401.

[20]

Hosmer D W Jr, Lemeshow S, Sturdivant RX (2013) Interpretation of the Fitted Logistic Regression Model. Wiley, pp 49-88.

[21]

Jiang Y, Cukic B, Menzies T (2008) Can data transformation help in the detection of fault-prone modules? In: Proceedings of the 2008 workshop on defects in large software systems, DEFECTS '08, pp 16-20.

[22]

Kim S, Zhang H, Wu R, Gong L (2011) Dealing with noise in defect prediction. In: Proceedings of the 33rd international conference on software engineering, ICSE '11, pp 481-490.

[23]

Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: a proposed framework and novel findings. IEEE Trans Softw Eng (TSE) 34(4):485-496.

Digital Library

[24]

Li M, Zhang H, Wu R, Zhou ZH (2012) Sample-based software defect prediction with active and semisupervised learning. Autom Softw Eng 19(2):201-230.

Digital Library

[25]

Ma Y, Luo G, Zeng X, Chen A (2012) Transfer learning for cross-company software defect prediction. Inf Softw Technol 54(3):248-256.

Digital Library

[26]

Mair C, Shepperd M (2005) The consistency of empirical comparisons of regression and analogy-based software project cost prediction. In: Proceedings of the 2005 international symposium on empirical software engineering, pp 509-518.

[27]

Menzies T, Dekhtyar A, Distefano J, Greenwald J (2007a) Problems with precision: a response to comments on 'data mining static code attributes to learn defect predictors. IEEE Trans Softw Eng (TSE) 33(9):637-640.

Digital Library

[28]

Menzies T, Greenwald J, Frank A (2007b) Data mining static code attributes to learn defect predictors. IEEE Trans Softw Eng (TSE) 33(1):2-13.

Digital Library

[29]

Menzies T, Butcher A, Marcus A, Zimmermann T, Cok D (2011) Local vs. global models for effort estimation and defect prediction. In: Proceedings of the 2011 26th IEEE/ACM international conference on automated software engineering, ASE '11, pp 343-351.

[30]

Mockus A (2009) Amassing and indexing a large sample of version control systems: towards the census of public source code history. In: Proceedings of the 6th IEEE international working conference on mining software repositories, MSR'09, pp 11-20.

[31]

Mockus A, Votta L (2000) Identifying reasons for software changes using historic databases. In: Proceedings of the 16th international conference on software maintenance, ICSM '00, pp 120-130.

[32]

Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering, vol 2013. ACM, New York, pp 466-476.

[33]

Nagappan N, Ball T, Zeller A (2006) Mining metrics to predict component failures. In: Proceedings of the 28th international conference on software engineering, ACM, ICSE '06, pp 452-461.

[34]

Nam J, Pan SJ, Kim S (2013) Transfer defect learning. In: Proceedings of the 2013 international conference on software engineering, ICSE '13, pp 382-391.

[35]

Nguyen TT, Nguyen TN, Phuong TM (2011) Topic-based defect prediction (nier track). In: Proceedings of the 33rd international conference on software engineering, ICSE '11. ACM, New York, pp 932-935.

[36]

Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345-1359.

Digital Library

[37]

Peters F, Menzies T, Gong L, Zhang H (2013a) Balancing privacy and utility in cross-company defect prediction. IEEE Trans Softw Eng 39(8):1054-1068.

Digital Library

[38]

Peters F, Menzies T, Marcus A (2013b) Better cross company defect prediction. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR '13, pp 409-418.

[39]

Posnett D, Filkov V, Devanbu P (2011) Ecological inference in empirical software engineering. In: Proceedings of the 26th IEEE/ACM international conference on automated software engineering, ASE '11. IEEE Computer Society, Washington, pp 362-371.

[40]

Premraj R, Herzig K (2011) Network versus code metrics to predict defects: a replication study. In: 2011 international symposium on empirical software engineering and measurement (ESEM), pp 215-224.

[41]

Radjenovic D, Heri¿ko M, Torkar R, ¿ivkovi¿ A (2013) Software fault prediction metrics: A systematic literature review. Inf Softw Technol 55(8):1397-1418.

Digital Library

[42]

Rahman F, Posnett D, Devanbu P (2012) Recalling the "imprecision" of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, FSE '12, pp 61:1-61:11.

[43]

Rahman F, Posnett D, Herraiz I, Devanbu P (2013) Sample size vs. bias in defect prediction. In: Proceedings of the 21th ACM SIGSOFT symposium and the 15th European conference on foundations of software engineering, ESEC/FSE '13.

[44]

Romano J, Kromrey JD, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: should we really be using t-test and cohen's d for evaluating group differences on the nsse and other surveys? In: Annual meeting of the Florida association of institutional research, pp 1-33.

[45]

Sarro F, Di Martino S, Ferrucci F, Gravino C (2012) A further analysis on the use of genetic algorithm to configure support vector machines for inter-release fault prediction. In: Proceedings of the 27th annual ACM symposium on applied computing, SAC '12. ACM, New York, pp 1215-1220.

[46]

SciTools (2015) Understand 3.1 build 726. https://scitools.com, [Online; accessed 15-June-2015].

[47]

Shatnawi R, Li W (2008) The effectiveness of software metrics in identifying error-prone classes in post-release software evolution process. J Syst Softw 81(11):1868-1882.

Digital Library

[48]

Sheskin DJ (2007) Handbook of parametric and nonparametric statistical procedures, 4th edn. Chapman & Hall/CRC.

[49]

Shihab E, Jiang ZM, Ibrahim WM, Adams B, Hassan AE (2010) Understanding the impact of code and process metrics on post-release defects: a case study on the eclipse project. In: Proceedings of the 2010 ACM/IEEE international symposium on empirical software engineering and measurement, ESEM '10. ACM, New York, pp 4:1-4:10.

[50]

Sliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes? In: Proceedings of the 2nd international workshop on mining software repositories, MSR '05, pp 1-5.

[51]

Tassey G (2002) The economic impacts of inadequate infrastructure for software testing. Tech. Rep. Planning Report 02-3, National Institute of Standards and Technology.

[52]

Turhan B, Menzies T, Bener AB, Di Stefano J (2009) On the relative value of cross-company and within company data for defect prediction. Empir Softw Eng 14(5):540-578.

Digital Library

[53]

Watanabe S, Kaiya H, Kaijiri K (2008) Adapting a fault prediction model to allow inter languagereuse. In: Proceedings of the 4th international workshop on predictor models in software engineering, PROMISE '08. ACM, New York, pp 19-24.

[54]

Yin RK (2002) Case study research: design and methods, 3rd edn. SAGE Publications.

[55]

Zhang F, Mockus A, Zou Y, Khomh F, Hassan AE (2013) How does context affect the distribution of software maintainability metrics? In: Proceedings of the 29th IEEE international conference on software maintainability, ICSM '13, pp 350-359.

[56]

Zhang F, Mockus A, Keivanloo I, Zou Y (2014) Towards building a universal defect prediction model. In: Proceedings of the 11th working conference on mining software repositories, MSR '14, pp 41-50.

[57]

Zhou Y, Leung H (2007) Predicting object-oriented software maintainability using multivariate adaptive regression splines. J Syst Softw 80(8):1349-1361.

Digital Library

[58]

Zimmermann T, Nagappan N (2008) Predicting defects using network analysis on dependency graphs. In: Proceedings of the 30th international conference on software engineering, ICSE '08. ACM, New York, pp 531-540.

[59]

Zimmermann T, Premraj R, Zeller A (2007) Predicting defects for eclipse. In: Proceedings of the international workshop on predictor models in software engineering, PROMISE '07, p 9.

[60]

Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. In: Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, ESEC/FSE '09, pp 91-100.

[61]

Zimmermann T, Nagappan N, Guo PJ, Murphy B (2012) Characterizing and predicting which bugs get reopened. In: 34th International Conference on Software Engineering (ICSE), 2012, pp 1074-1083.

Digital Library

Cited By

Wang YLi YWang HZhao LZhang X(2025)Better Knowledge Enhancement for Privacy‐Preserving Cross‐Project Defect PredictionJournal of Software: Evolution and Process10.1002/smr.276137:1Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1002/smr.2761
Zhao GGeorgiou SHassan SZou YTruong DCorbin TSpinellis DConstantinou EBacchelli A(2024)Enhancing Performance Bug Prediction Using Performance Code MetricsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644920(50-62)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644920
Tong HZhang DLiu JXing WLu LLu WWu Y(2024)MASTER: Multi-Source Transfer Weighted Ensemble Learning for Multiple Sources Cross-Project Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2024.338123550:5(1281-1305)Online publication date: 25-Mar-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3381235
Show More Cited By

Recommendations

Towards building a universal defect prediction model
MSR 2014: Proceedings of the 11th Working Conference on Mining Software Repositories

To predict files with defects, a suitable prediction model must be built for a software project from either itself (within-project) or other projects (cross-project). A universal defect prediction model that is built from the entire set of diverse ...
Cross-project smell-based defect prediction
Abstract
Defect prediction is a technique introduced to optimize the testing phase of the software development pipeline by predicting which components in the software may contain defects. Its methodology trains a classifier with data regarding a set of ...
Assessing Software Quality by Program Clustering and Defect Prediction
WCRE '11: Proceedings of the 2011 18th Working Conference on Reverse Engineering

Many empirical studies have shown that defect prediction models built on product metrics can be used to assess the quality of software modules. So far, most methods proposed in this direction predict defects by class or file. In this paper, we propose a ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 21, Issue 5

October 2016

389 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2016 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 October 2016

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang YLi YWang HZhao LZhang X(2025)Better Knowledge Enhancement for Privacy‐Preserving Cross‐Project Defect PredictionJournal of Software: Evolution and Process10.1002/smr.276137:1Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1002/smr.2761
Zhao GGeorgiou SHassan SZou YTruong DCorbin TSpinellis DConstantinou EBacchelli A(2024)Enhancing Performance Bug Prediction Using Performance Code MetricsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644920(50-62)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644920
Tong HZhang DLiu JXing WLu LLu WWu Y(2024)MASTER: Multi-Source Transfer Weighted Ensemble Learning for Multiple Sources Cross-Project Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2024.338123550:5(1281-1305)Online publication date: 25-Mar-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3381235
Majumder SChakraborty JMenzies T(2024)When less is more: on the value of “co-training” for semi-supervised software defect predictorsEmpirical Software Engineering10.1007/s10664-023-10418-429:2Online publication date: 24-Feb-2024
https://dl.acm.org/doi/10.1007/s10664-023-10418-4
Manchala PBisi M(2024)A study on cross-project fault prediction through resampling and feature reduction along with source projects selectionAutomated Software Engineering10.1007/s10515-024-00465-631:2Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1007/s10515-024-00465-6
C. SMenzies T(2023)Assessing the Early Bird Heuristic (for Predicting Project Quality)ACM Transactions on Software Engineering and Methodology10.1145/358356532:5(1-39)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3583565
Zhao YDamevski KChen H(2023)A Systematic Survey of Just-in-Time Software Defect PredictionACM Computing Surveys10.1145/356755055:10(1-35)Online publication date: 2-Feb-2023
https://dl.acm.org/doi/10.1145/3567550
Tong HLu WXing WWang S(2023)ARRAYJournal of Systems and Software10.1016/j.jss.2023.111721202:COnline publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111721
Alweshah MKassaymeh SAlkhalaileh SAlmseidin MAltarawni I(2023)An Efficient Hybrid Mine Blast Algorithm for Tackling Software Fault Prediction ProblemNeural Processing Letters10.1007/s11063-023-11357-355:8(10925-10950)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s11063-023-11357-3
Falessi DLaureani SÇarka JEsposito MCosta D(2023)Enhancing the defectiveness prediction of methods and classes via JITEmpirical Software Engineering10.1007/s10664-022-10261-z28:2Online publication date: 31-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10261-z
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents