research-article

Validation of network measures as indicators of defective modules in software systems

Authors:

Ayşe Tosun,

Burak Turhan,

Ayşe BenerAuthors Info & Claims

PROMISE '09: Proceedings of the 5th International Conference on Predictor Models in Software Engineering

Article No.: 5, Pages 1 - 9

https://doi.org/10.1145/1540438.1540446

Published: 18 May 2009 Publication History

Get Access

Abstract

In ICSE'08, Zimmermann and Nagappan show that network measures derived from dependency graphs are able to identify critical binaries of a complex system that are missed by complexity metrics. The system used in their analysis is a Windows product. In this study, we conduct additional experiments on public data to reproduce and validate their results. We use complexity and network metrics from five additional systems. We examine three small scale embedded software and two versions of Eclipse to compare defect prediction performance of these metrics. We select two different granularity levels to perform our experiments: function-level and source file-level. In our experiments, we observe that network measures are important indicators of defective modules for large and complex systems, whereas they do not have significant effects on small scale projects.

References

[1]

Zimmermann, T. and Nagappan, N. 2008. Predicting Defect Using Network Analysis on Dependency Graphs. In Proceedings of the International Conference on Software Engineering (ICSE'08), Leipzig, Germany, 531--540.

Digital Library

Google Scholar

[2]

Tosun, A., Turhan, B. and Bener, A. 2008. Ensemble of Software Defect Predictors: A Case Study. In Proceedings of the International Conference on Empirical Software Engineering and Measurement (ESEM), Keiserslautern, Germany, 318--320.

Digital Library

Google Scholar

[3]

Adrion, R. W., Branstad, A. M. and Cherniavsky, C. J. 1982. Validation, Verification and Testing of Computer Software. ACM Computing Surveys, June 1982, Vol. 14, No. 2, 159--192.

Digital Library

Google Scholar

[4]

Menzies, T., Greenwald, J. and Frank, A. 2007. Data Mining Static Code Attributes to Learn Defect Predictors. IEEE Transactions on Software Engineering, January 2007, Vol. 33, No. 1, 2--13.

Digital Library

Google Scholar

[5]

Mendonca, M. G., Maldonado, J. C., De Oliveira, M. C. F., Carver, J., Fabbri, S. C. P. F., Shull, F., Travassos, G. H., Hohn, E. N. and Basili, V. R. 2008. A Framework for Software Engineering Experimental Replications. 13th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS 2008), 203--212.

Digital Library

Google Scholar

[6]

Challagula, U. B. V., Bastani, B. F., Yen, L. and Paul, A. R. 2005. Empirical Assessment of Machine Learning based Software Defect Prediction Techniques. In Proceedings of the 10th IEEE International Workshop on Object-Oriented Real Time Dependable Systems, Sedona, USA, 2--4 February 2005, 263--270.

Digital Library

Google Scholar

[7]

Fenton, E. N. and Neil, M. 1999. A Critique of Software Defect Prediction Models. IEEE Transactions on Software Engineering, September/October 1999, Vol. 25, No. 5, 675--689.

Digital Library

Google Scholar

[8]

Turhan, B. and Bener, A. 2009. Analysis of Naive Bayes' Assumptions on Software Fault Data: An Empirical Study. Data and Knowledge Engineering Journal, Vol. 68, No. 2, 78--290.

Digital Library

Google Scholar

[9]

Ostrand, T. J., Weyuker, E. J. and Bell, R. M. 2005. Predicting the Location and Number of Faults in Large Software Systems. IEEE Transactions on Software Engineering, April 2005, Vol. 31, No. 4, 340--355.

Digital Library

Google Scholar

[10]

Halstead, M. H. 1977. Elements of Software Science, Elsevier, New York.

Digital Library

Google Scholar

[11]

McCabe, T. 1976. A Complexity Measure. IEEE Transactions on Software Engineering, Vol. 2, No. 4, 308--320.

Digital Library

Google Scholar

[12]

Fagan, M. 1976. Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal, Vol. 15, No. 3, 182--211.

Digital Library

Google Scholar

[13]

Nagappan, N. and Ball, T. 2005. Static Analysis Tools as Early Indicators of Pre-Release Defect Density. In Proceedings of the International Conference on Software Engineering (ICSE), St. Louise, USA, 580--586.

Digital Library

Google Scholar

[14]

Nagappan, N. and Ball, T. 2007. Using Software Dependencies and Churn Metrics to Predict Field Failures: An Empirical Case Study. In Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM), Madrid, Spain, 364--373.

Digital Library

Google Scholar

[15]

Nagappan, N. and Ball, T. 2005. Use of Relative Code Churn Measures to Predict System Defect Density. In Proceedings of the International Conference on Software Engineering (ICSE), St. Louis, USA, 284--292.

Digital Library

Google Scholar

[16]

Nagappan, N., Murphy, B. and Basili, V. R. 2008. The Influence of Organizational Structure on Software Quality: An Empirical Case Study. In Proceedings of the International Conference on Software Engineering (ICSE), Leipzig, Germany, 521--530.

Digital Library

Google Scholar

[17]

Hall, M. and Holmes, G. 2003. Benchmarking Attribute Selection Techniques for Discrete Class Data Mining. IEEE Transactions on Knowledge and Data Engineering, Vol. 15, No. 6, 1437--1447.

Digital Library

Google Scholar

[18]

Menzies, T., Turhan, B., Bener, A., Gay, G., Cukic, B. and Jiang, Y. 2008. Implications of Ceiling Effects in Defect Predictors. In Promise Workshop (part of the 30th International Conference on Software Engineering), Germany, 47--54.

Digital Library

Google Scholar

[19]

Turhan, B., Menzies, T., Bener, A., Distefano, J. 2008. On the Relative Value of Cross-company and Within-Company Data for Defect Prediction. accepted for publication in Empirical Software Engineering Journal.

Digital Library

Google Scholar

[20]

Boetticher, G., Menzies, T. and Ostrand, T. 2007. PROMISE Repository of empirical software engineering data http://promisedata.org/repository, West Virginia University, Department of Computer Science.

Google Scholar

[21]

Eclipse project archived downloads page: http://archive.eclipse.org/eclipse/downloads

Google Scholar

[22]

Ucinet tool download page: http://www.analytictech.com/ucinet/ucinet.htm.

Google Scholar

[23]

Prest. 2009. Department of Computer Engineering, Bogazici University, http://code.google.com/p/prest/.

Google Scholar

[24]

Schröder, A., Zimmermann, T., Premraj, R. and Zeller, A. 2006. If Your Bug Database Could Talk. In Proceedings of the International Symposium on Empirical Software Engineering (ISESE), Brazil, 18--20.

Google Scholar

[25]

Alpaydin, E. 2004. Introduction to Machine Learning, MIT Press, Massachusetts.

Digital Library

Google Scholar

[26]

Turhan, B., Bener, A. and Kocak, G. 2008. Data Mining Source Code for Locating Software Bugs: A Case Study in Telecommunication Industry. accepted for publication in Expert Systems with Applications Journal.

Digital Library

Google Scholar

[27]

Kocak, G. 2008. Software Defect Prediction Using Call Graph Based Ranking (CGBR) Framework. MS Thesis, Boğaziçi University, Turkey.

Google Scholar

[28]

Brin, S. and Page, L. 1998. The anatomy of a large-scale hypertextual search engine. Computer Networks and ISDN Systems, 107--117.

Digital Library

Google Scholar

[29]

Heeger, D. 1998. Signal Detection Theory. http://www.cns.nyu.edu/~david/handouts/sdt/sdt.html.

Google Scholar

[30]

Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K., and Rosenberg, J. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, Vol. 28, No. 8, 721--734.

Digital Library

Google Scholar

Cited By

View all

Fehrer TCabrera Lozoya RSabetta ADi Nucci DTamburri D(2024)Detecting Security Fixes in Open-Source Repositories using Static Code AnalyzersProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661217(429-432)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661217
Nabi FZhou XGururajan R(2024)An enhanced conventional neural network schema for structural class-based software fault predictionJournal of Cyber Security Technology10.1080/23742917.2024.2402014(1-22)Online publication date: 12-Sep-2024
https://doi.org/10.1080/23742917.2024.2402014
Levasseur MBadri M(2024)Prioritizing unit tests using object-oriented metrics, centrality measures, and machine learning algorithmsInnovations in Systems and Software Engineering10.1007/s11334-024-00550-9Online publication date: 2-Feb-2024
https://doi.org/10.1007/s11334-024-00550-9
Show More Cited By

Index Terms

Validation of network measures as indicators of defective modules in software systems
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation
      2. Process validation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software performance
    2. Software functional properties
      1. Formal methods

Recommendations

Network Versus Code Metrics to Predict Defects: A Replication Study
ESEM '11: Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement

Several defect prediction models have been proposed to identify which entities in a software system are likely to have defects before its release. This paper presents a replication of one such study conducted by Zimmermann and Nagappan on Windows Server ...
Different strokes for different folks: a case study on software metrics for different defect categories
WETSoM '11: Proceedings of the 2nd International Workshop on Emerging Trends in Software Metrics

Defect prediction has been evolved with variety of metric sets, and defect types. Researchers found code, churn, and network metrics as significant indicators of defects. However, all metric sets may not be informative for all defect categories such ...
Defect prediction using social network analysis on issue repositories
ICSSP '11: Proceedings of the 2011 International Conference on Software and Systems Process

People are the most important pillar of software development process. It is critical to understand how they interact with each other and how these interactions affect the quality of the end product in terms of defects. In this research we propose to ...

Reviews

Reviewer: David A. Gustafson

The ability to identify software modules that contain defects by using network measures would be a tremendous achievement in software engineering. This paper reproduces the results of Zimmermann and Nagappan [1] in using network measures to predict error-prone modules. This result is significant. Although complexity measures have been the standard basis for prediction models, this paper shows that the use of dynamic-call measures significantly improves the predictive ability. What the paper does not do is validate the specific prediction model of Zimmermann and Nagappan or the strength of the correlations that Zimmermann and Nagappan found between the specific network measures and the defects. Thus, we are left with the knowledge that in different software systems, some network measures will significantly improve the predictive ability of an error-prediction model, but not necessarily the same ones. Tosun, Turhan, and Bener show that it is possible to generate a prediction model based on some complexity measures and some network measures that is effective in predicting which modules are error prone; however, they do not show that the same prediction model will work in different systems. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

PROMISE '09: Proceedings of the 5th International Conference on Predictor Models in Software Engineering

May 2009

268 pages

ISBN:9781605586342

DOI:10.1145/1540438

Conference Chair:
Tom Ostrand
AT&T USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 May 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

Promise '09

Promise '09: 5th International Workshop on Predictor Models in SE

May 18 - 19, 2009

British Columbia, Vancouver, Canada

Acceptance Rates

Overall Acceptance Rate 98 of 213 submissions, 46%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
532
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)1

Reflects downloads up to 06 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fehrer TCabrera Lozoya RSabetta ADi Nucci DTamburri D(2024)Detecting Security Fixes in Open-Source Repositories using Static Code AnalyzersProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering10.1145/3661167.3661217(429-432)Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1145/3661167.3661217
Nabi FZhou XGururajan R(2024)An enhanced conventional neural network schema for structural class-based software fault predictionJournal of Cyber Security Technology10.1080/23742917.2024.2402014(1-22)Online publication date: 12-Sep-2024
https://doi.org/10.1080/23742917.2024.2402014
Levasseur MBadri M(2024)Prioritizing unit tests using object-oriented metrics, centrality measures, and machine learning algorithmsInnovations in Systems and Software Engineering10.1007/s11334-024-00550-9Online publication date: 2-Feb-2024
https://doi.org/10.1007/s11334-024-00550-9
Gong LZhang HZhang JWei MHuang Z(2023)A Comprehensive Investigation of the Impact of Class Overlap on Software Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2022.322074049:4(2440-2458)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3220740
Qin FZheng ZWan XLiu ZShi Z(2023)Predicting Aging-Related Bugs Using Network Analysis on Aging-Related Dependency NetworksIEEE Transactions on Emerging Topics in Computing10.1109/TETC.2023.327938811:3(566-579)Online publication date: 1-Jul-2023
https://doi.org/10.1109/TETC.2023.3279388
Ouellet ABadri M(2023)Combining object‐oriented metrics and centrality measures to predict faults in object‐oriented softwareJournal of Software: Evolution and Process10.1002/smr.254836:4Online publication date: 9-Mar-2023
https://dl.acm.org/doi/10.1002/smr.2548
Du XWang TWang LPan WChai CXu XJiang BWang J(2022)CoreBug: Improving Effort-Aware Bug Prediction in Software Systems Using Generalized k-Core Decomposition in Class Dependency NetworksAxioms10.3390/axioms1105020511:5(205)Online publication date: 27-Apr-2022
https://doi.org/10.3390/axioms11050205
Huch F(2022)Formal Entity Graphs as Complex Networks: Assessing Centrality Metrics of the Archive of Formal ProofsIntelligent Computer Mathematics10.1007/978-3-031-16681-5_10(147-161)Online publication date: 17-Sep-2022
https://doi.org/10.1007/978-3-031-16681-5_10
Gong LRajbahadur GHassan AJiang S(2021)Revisiting the Impact of Dependency Network Metrics on Software Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2021.3131950(1-1)Online publication date: 2021
https://doi.org/10.1109/TSE.2021.3131950
Qu YZheng QChi JJin YHe ACui DZhang HLiu T(2021)Using K-core Decomposition on Class Dependency Networks to Improve Bug Prediction Model's Practical PerformanceIEEE Transactions on Software Engineering10.1109/TSE.2019.289295947:2(348-366)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1109/TSE.2019.2892959
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Network Versus Code Metrics to Predict Defects: A Replication Study

Different strokes for different folks: a case study on software metrics for different defect categories

Defect prediction using social network analysis on issue repositories

Reviews

Access critical reviews of Computing literature here

Comments

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Network Versus Code Metrics to Predict Defects: A Replication Study

Different strokes for different folks: a case study on software metrics for different defect categories

Defect prediction using social network analysis on issue repositories

Reviews

Access critical reviews of Computing literature here

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations