research-article

Validating software metrics: A spectrum of philosophies

Authors:

Andrew Meneely,

Laurie WilliamsAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 21, Issue 4

Article No.: 24, Pages 1 - 28

https://doi.org/10.1145/2377656.2377661

Published: 07 February 2013 Publication History

Abstract

Context. Researchers proposing a new metric have the burden of proof to demonstrate to the research community that the metric is acceptable in its intended use. This burden of proof is provided through the multi-faceted, scientific, and objective process of software metrics validation. Over the last 40 years, however, researchers have debated what constitutes a “valid” metric.

Aim. The debate over what constitutes a valid metric centers on software metrics validation criteria. The objective of this article is to guide researchers in making sound contributions to the field of software engineering metrics by providing a practical summary of the metrics validation criteria found in the academic literature.

Method. We conducted a systematic literature review that began with 2,288 papers and ultimately focused on 20 papers. After extracting 47 unique validation criteria from these 20 papers, we performed a comparative analysis to explore the relationships amongst the criteria.

Results. Our 47 validation criteria represent a diverse view of what constitutes a valid metric. We present an analysis of the criteria's categorization, conflicts, common themes, and philosophical motivations behind the validation criteria.

Conclusions. Although the 47 validation criteria are not conflict-free, the diversity of motivations and philosophies behind the validation criteria indicates that metrics validation is complex. Researchers proposing new metrics should consider the applicability of the validation criteria in terms of our categorization and analysis. Rather than arbitrarily choosing validation criteria for each metric, researchers should choose criteria that can confirm that the metric is appropriate for its intended use. We conclude that metrics validation criteria provide answers to questions that researchers have about the merits and limitations of a metric.

References

[1]

Ayewah, N., Hovemeyer, D., Morgenthaler, J. D., Penix, J., and Pugh, W. 2008. Using Static Analysis to Find Bugs.IEEE Softw. 25, 5, 22--29.

Digital Library

[2]

Baker, A. L., Bieman, J. M., Fenton, N., Gustafson, D., Melton, A., and Whitty, R. 1990. A philosophy for software measurement. J. Syst. Softw. 12, 3, 277--281.

Digital Library

[3]

Basili, V. R. and Weiss, M. 1994. A methodology for collecting valid software engineering data. IEEE Trans. Softw. Engin. SE-IO, 6, 728--738.

Digital Library

[4]

Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M. 2007. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80, 571--583.

Digital Library

[5]

Briand, L., Emam, K. E., and Morasca, S. 1996. On the Application of Measurement Theory in Software Engineering. Empirical Softw. Engin. 1, 1, 61--88.

[6]

Briand, L., Emam, K. E., and Morasca, S. 1995. Theoretical and empirical validation of software product measures. Tech. rep. &num;ISERN-95-04. International Software Engineering Research Network.

[7]

Bush, M. E. and Fenton, N. E. 1990. Software measurement: A conceptual framework. J. Syst. Softw. 12, 3, 223--231.

Digital Library

[8]

Cavano, J. and McCall, J. 1978. A framework for the measurement of software quality. In Proceedings of the Software Quality Assurance Workshop on Functional and Performance Issues.

Digital Library

[9]

Courtney, R. E. and Gustafson, D. A. 1993. Shotgun correlations in software measures. Softw. Engin. J. 8, 1, 5--13.

Digital Library

[10]

Curtis, B. 1980. Measurement and experimentation in software engineering. Proc. IEEE 68, 9, 1144--1157.

[11]

El Emam, K. 2000. A methodology for validating software product metrics. Tech. rep. NCR/ERC-1076, National Research Council of Canada, Ottawa, Ontario, Canada.

[12]

Elbaum, S. G. and Munson, J. C. 1998. Getting a handle on the fault injection process: validation of measurement tools. In Proceedings of the 5th International Software Metrics Symposium.

Digital Library

[13]

Fenton, N. 1994. Software measurement: a necessary scientific basis. IEEE Trans. Softw. Engin. 20, 3, 199--206.

Digital Library

[14]

Fenton, N. and Kitchenham, B. 1991. Validating software measures. J. Softw. Testing Verif. Reliab. 1, 2, 27--42.

Digital Library

[15]

Fenton, N. and Neil, M. 1999. Software metrics: successes, failures and new directions. J. Syst. Softw. 47, 2--3, 149--157.

Digital Library

[16]

Fenton, N. and Neil, M. 2000. Software metrics: roadmap. Future of Software Engineering. In Proceedings of the Conference on The Future of Software Engineering.

Digital Library

[17]

Harman, M. and Clark, J. 2004. Metrics are fitness functions too. In Proceedings of the 10th International Symposium on Software Metrics.

Digital Library

[18]

Henderson-Sellers, B. 1996. The mathematical validity of software metrics. ACM SIGSOFT Softw. Engin. Notes 21, 5, 89--94.

Digital Library

[19]

IEEE 1990. IEEE Standard 610.12-1990. IEEE Standard Glossary of Software Engineering Terminology.

[20]

ISO/IEC 1991. ISO/IEC 9126: Information technology - software product evaluation - quality characteristics and guidelines for their use.

[21]

Jones, C. 1994. Software metrics: good, bad and missing. Computer 27, 9, 98--100.

Digital Library

[22]

Kitchenham, B. 2004. Procedures for performing systematic reviews. Joint Tech. rep., Keele University Tech. rep. TR/SE-0401 and NICTA Tech. rep. 0400011T.

[23]

Kitchenham, B. 2010. What's up with software metrics&quest; A preliminary mapping study. J. Syst. Softw. 83, 1, 37--51.

Digital Library

[24]

Kitchenham, B., Pfleeger, S. L., and Fenton, N. 1995. Towards a framework for software measurement validation. IEEE Trans. Softw. Engin. 21, 12, 929--944.

Digital Library

[25]

Lincke, R. and Lowe, W. 2006. Foundations for defining software metrics. In Proceedings of the 3rd International Workshop on Metamodels, Schemas, Grammars, and Ontologies for Reverse Engineering (ATEM'06).

[26]

McCabe, T. 1976. A complexity measure. IEEE Trans. Softw. Engin. SE-2, 4, 308--320.

Digital Library

[27]

Rao, P. V. 2007. Statistical Research Methods in Life Sciences. Duxbury Press, Pacific Grove, CA.

[28]

Roche, J. M. 1994. Software metrics and measurement principles. ACM SIGSOFT Softw. Engin. Notes 19, 1, 77--85.

Digital Library

[29]

Schneidewind, N. F. 1991. Validating software metrics: producing quality discriminators. In Proceedings of the International Symposium on Software Reliability Engineering.

[30]

Schneidewind, N. F. 1992. Methodology for validating software metrics. IEEE Trans. Softw. Engin. 18, 5, 410--422.

Digital Library

[31]

Weyuker, E. J. 1988. Evaluating software complexity measures. IEEE Trans. Softw. Engin. 14, 9, 1357--1365.

Digital Library

Cited By

Gyeong Kim MChang Lee K(2025)Proposing the “Digital Agenticity Theory” to analyze user engagement in conversational AI chatbotJournal of Business Research10.1016/j.jbusres.2024.115162189(115162)Online publication date: Feb-2025
https://doi.org/10.1016/j.jbusres.2024.115162
Connolly Bree DÓ Cinnéide MCai YChaudron MKang Evan der Hoek A(2024)Weighted Metrics for the Development of Energy Efficient SoftwareProceedings of the 1st International Workshop on Designing Software10.1145/3643660.3643946(64-69)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643660.3643946
Ofem PIsong BLugayizi F(2024)Defining Metrics for Evaluating Transparency in Software Engineering Process2024 14th International Conference on Software Technology and Engineering (ICSTE)10.1109/ICSTE63875.2024.00009(1-7)Online publication date: 16-Aug-2024
https://doi.org/10.1109/ICSTE63875.2024.00009
Show More Cited By

Index Terms

Validating software metrics: A spectrum of philosophies
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
    2. Software verification and validation
      1. Process validation

Recommendations

Methodology for Validating Software Metrics

A comprehensive metrics validation methodology is proposed that has six validity criteria, which support the quality functions assessment, control, and prediction, where quality functions are activities conducted by software organizations for the ...
Software fault prediction metrics

ContextSoftware metrics may be used in fault prediction models to improve software quality by predicting fault location. ObjectiveThis paper aims to identify software metrics and to assess their applicability in software fault prediction. We ...
Adoption of object-oriented software metrics for ontology evaluation
BCI '12: Proceedings of the Fifth Balkan Conference in Informatics

Object-oriented software metrics are well established and widely acknowledged as a measure of software quality. The aim of our research is to analyze the potential use of some of these metrics for ontology evaluation. In this paper we present the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 21, Issue 4

November 2012

197 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2377656

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2013

Accepted: 01 June 2011

Revised: 01 May 2011

Received: 01 September 2010

Published in TOSEM Volume 21, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

50
Total Citations
View Citations
1,776
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)12

Reflects downloads up to 04 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gyeong Kim MChang Lee K(2025)Proposing the “Digital Agenticity Theory” to analyze user engagement in conversational AI chatbotJournal of Business Research10.1016/j.jbusres.2024.115162189(115162)Online publication date: Feb-2025
https://doi.org/10.1016/j.jbusres.2024.115162
Connolly Bree DÓ Cinnéide MCai YChaudron MKang Evan der Hoek A(2024)Weighted Metrics for the Development of Energy Efficient SoftwareProceedings of the 1st International Workshop on Designing Software10.1145/3643660.3643946(64-69)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643660.3643946
Ofem PIsong BLugayizi F(2024)Defining Metrics for Evaluating Transparency in Software Engineering Process2024 14th International Conference on Software Technology and Engineering (ICSTE)10.1109/ICSTE63875.2024.00009(1-7)Online publication date: 16-Aug-2024
https://doi.org/10.1109/ICSTE63875.2024.00009
Feitelson D(2023)From Code Complexity Metrics to Program ComprehensionCommunications of the ACM10.1145/354657666:5(52-61)Online publication date: 21-Apr-2023
https://dl.acm.org/doi/10.1145/3546576
Stolp FGrundy J(2023)Assessing Cognitive Load in Software Development with Wearable SensorsProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00062(227-229)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00062
Tahir TGencel CRasool GUmer TRasheed JYeo SCevik T(2023)Early Software Defects Density Prediction: Training the International Software Benchmarking Cross Projects Data Using Supervised LearningIEEE Access10.1109/ACCESS.2023.333999411(141965-141986)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3339994
Zhao XGray J(2023)Towards a metrics suite for the complexity analysis of LabVIEW systems modelsScience of Computer Programming10.1016/j.scico.2023.102931227(102931)Online publication date: Apr-2023
https://doi.org/10.1016/j.scico.2023.102931
Rani PBlasi AStulova NPanichella SGorla ANierstrasz O(2023)A decade of code comment quality assessmentJournal of Systems and Software10.1016/j.jss.2022.111515195:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111515
Wang LLi YZhang JTao XRoychoudhury ACadar CKim M(2022)Quantifying community evolution in developer social networksProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549106(157-169)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549106
Ofem PIsong BLugayizi F(2022)On the Concept of Transparency: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2022.320048710(89887-89914)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3200487
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents