Article

Evaluating the quality of information models: empirical testing of a conceptual model quality framework

Authors:

Daniel L. Moody,

Guttorm Sindre,

Terje Brasethvik,

Arne SølvbergAuthors Info & Claims

ICSE '03: Proceedings of the 25th International Conference on Software Engineering

Pages 295 - 305

Published: 03 May 2003 Publication History

Publisher Site Get Access

Abstract

This paper conducts an empirical analysis of a semiotics-based quality framework for quality assuring information models. 192 participants were trained in the concepts of the quality framework, and used it to evaluate models represented in an extended Entity Relationship (ER) language. A randomised, double-blind design was used, in which each participant independently reviewed multiple models and each model was evaluated by multiple reviewers. A combination of quantitative and qualitative analysis techniques were used to evaluate the results, including reliability analysis, validity analysis, interaction analysis, influence analysis, defect pattern analysis and task accuracy analysis. An analysis was also conducted of the framework's likelihood of adoption in practice. The study provides strong support for the validity of the framework and suggests that it is likely to be adopted in practice, but raises questions about its reliability and the ability of participants to use it to accurately identify defects. The research findings provide clear directions for improvement of the framework. The research methodology used provides a general approach to empirical validation of quality frameworks.

References

[1]

Alexander, C. (1968): Notes On The Synthesis Of Form, Boston: Harvard University Press.]]

[2]

Babbie, E. R. (1998): The Practice of Social Research, Belmont, California: Wadsworth Publishing.]]

[3]

Batra, C., S. Ceri, and S. B. Navathe (1992): Conceptual Database Design: An Entity Relationship Approach, Redwood City, California: Benjamin Cummings.]]

Digital Library

[4]

Batra, D., J. A. Hoffer, and R. P. Bostrom (1990): Comparing Representations with Relational and EER Models, Communications of the ACM, 33 (2), February: p. 126--139.]]

Digital Library

[5]

Bock, D. B. and T. Ryan (1993): Accuracy in Modelling with Extended Entity Relationship and Object Oriented Data Model, Journal Of Database Management, 4 (4): p. 30--39.]]

[6]

Bodart, F., A. Patel, M. Sim, and R. Weber (2001): Should the Optional Property Construct be used in Conceptual Modelling: A Theory and Three Empirical Tests, Information Systems Research, 12 (4).]]

Digital Library

[7]

Brasethvik, T. and J. A. Gulla (2002): A Conceptual Modeling Approach to Semantic Document Retrieval, In A. Banks Pidduck (Ed.) 14th International Conference on Advanced Information Systems Engineering (CAiSE'02), Toronto: Springer Verlag, May 27--31.]]

Digital Library

[8]

Bubenko, J. A. (1986): Information Systems Methodologies - A Research View, In Information Systems Design Methodologies: Improving The Practice, T. W. Olle, H. G. Sol, and A. A. Verrijn-Stuart (Eds.), North-Holland.]]

Digital Library

[9]

Chau, P. Y. K. (1996): An Empirical Assessment of a Modified Technology Acceptance Model, Journal of Management Information Systems, 13 (2).]]

Digital Library

[10]

Cohen, J. (1988): Statistical Power Analysis for the Behavioural Sciences (2nd ed.), 2nd ed, Hillsdale, NJ: Lawrence Earlbaum and Associates.]]

[11]

Cohen, P. A. (1981): Student Ratings of Instruction and Student Achievement: A Meta-Analysis of Multi-Section Validity Studies, Review of Educational Research, 51: p. 281--309.]]

[12]

Cooper, D. R. and P. S. Sehindler (1998): Business Research Methods (6th edition), 6th ed, Singapore: McGraw-Hill International.]]

[13]

Curtis, B. (1986): By The Way, Did Anyone Study Any Real Programmers?, In Empirical Studies of Programmers, E. Soloway and S. Iyengar (Eds.), Ablex: Norward, N. J.]]

Digital Library

[14]

Davis, F. D. (1989): Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information Technology, MIS Quarterly, 13 (3).]]

[15]

Davis, F. D., R. P. Bagozzi, and P. R. Warshaw (1989): User Acceptance of Computer Technology: A Comparison of Two Theoretical Models, Management Science, 35 (8): p. 982--1003.]]

Digital Library

[16]

Deming, W. E. (1986): Out of the Crisis, Cambridge, MA: MIT Center for Advanced Engineering.]]

[17]

Evans, J. R. and W. M. Lindsay (2002): The Management and Control of Quality (5th Ed.), Cincinnati, Ohio: South-Western (Thomson Learning).]]

[18]

Fitzgerald, G. (1991): Validating New Information Systems Techniques: A Retrospective Analysis, In H. E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches And Emergent Traditions: North-Holland,]]

[19]

Galliers, R. D. (1994): Relevance and Rigour in Information Systems Research: Some Personal Reflections on Issues Facing the Information Systems Research Community, In Proceedings of the IFIP TC8 Conference on Business Process Reengineering: Information Systems and Challenges, Gold Coast, Australia,]]

Digital Library

[20]

Gulla, J. A. and T. Brasethvik (2000): On the Challenges of Business Modelling in Large-scale Reengineering Projects, In 4th International Conference on Requirements Engineering (ICRE 2000), Schaumburg, IL: IEEE Computer Society Press, 19--23 June 2000.]]

Digital Library

[21]

Hardgrave, B. C. and N. P. Dalal (1995): Comparing Object Oriented and Extended Entity Relationship Models, Journal Of Database Management, 6 (3).]]

[22]

Harris, M. J. and R. Rosenthal (1985): Mediation of Interpersonal Expectancy Effects: 31 Meta-Analyses, Psychological Bulletin, 97.]]

[23]

Hu, P. J. and P. Y. K. Chau (1999): Examining the Technology Acceptance Model Using Physician Acceptance of Telemedicine Technology, Journal of Management Information Systems, 16 (2), Fall: p. 91--113.]]

Digital Library

[24]

Ivari, J. (1986): Dimensions Of Information Systems Design: A Framework For A Long Range Research Program, Information Systems Journal, (June).]]

Digital Library

[25]

Kaplan, B. and D. Duchon (1988): Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study, MIS Quarterly, December.]]

Digital Library

[26]

Keen, P. G. W. (1991): Relevance and Rigour in Information Systems Research: Improving Quality, Confidence, Cohesion and Impact, In Information Systems Research: Contemporary Approaches and Emergent Traditions, H.-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Elsevier Science Publishers (North Holland).]]

[27]

Kesh, S. (1995): Evaluating The Quality Of Entity Relationship Models, Information And Software Technology, 37 (12)]]

[28]

Krogstie, J., O. I. Lindland, and G. Sindre (1995): Towards a Deeper Understanding of Quality in Requirements Engineering, In Proceedings of the 7th International Conference on Advanced Information Systems Engineering (CAISE), Jyvaskyla, Finland, June.]]

Digital Library

[29]

Krogstie, J. (1998): Integrating the Understanding of Quality in Requirements Specification and Conceptual Modelling, Software Engineering Notes, 28 (1).]]

Digital Library

[30]

Lauesen, S. and O. Vinter (2000): Preventing Requirement Defects, In Proceedings of the Sixth International Workshop on Requirements Engineering: Foundation for Software Quality (REFSQ'2000), Stockholm, Sweden, June 5--6 2000.]]

[31]

Lee, H. and B. G. Choi (1998): A Comparative Study of Conceptual Data Modelling Techniques, Journal Of Database Management, 9 (2), Spring: p. 26--35.]]

Digital Library

[32]

Levitin, A. and T. Redman (1994): Quality Dimensions of a Conceptual View, Information Processing and Management, 31 (1).]]

Digital Library

[33]

Lindland, O. I., G. Sindre, and A. Sølvberg (1994): Understanding Quality In Conceptual Modelling, IEEE Software, 11 (2), March: p. 42--49.]]

Digital Library

[34]

Martin, J. (1989): Information Engineering, Englewood Cliffs, N. J.: Prentice Hall, 3 v.]]

Digital Library

[35]

Moody, D. L. and G. G. Shanks (1994): What Makes A Good Data Model? Evaluating the Quality of Entity Relationship Models, In P. Loucopolous (Ed.) Proceedings of the Thirteenth International Conference on the Entity Relationship Approach, Manchester, England, December 14--17.]]

Digital Library

[36]

Moody, D. L. and G. G. Shanks (1998): Evaluating and Improving the Quality of Entity Relationship Models: An Action Research Programme, Australian Computer Journal, November.]]

[37]

Moody, D. L. and G. G. Shanks (1998): What Makes A Good Data Model? A Framework For Evaluating And Improving The Quality Of Entity Relationship Models, Australian Computer Journal, August.]]

[38]

Moody, D. L. (2000): Building Links Between IS Research and Professional Practice: Improving the Relevance and Impact of IS Research, In R. A. Weber and B. Glasson (Eds.), International Conference on Information Systems (ICIS'00), Brisbane, Australia, December 11--13.]]

Digital Library

[39]

Moody, D. L. (2001): Dealing with Complexity: A Practical Method for Representing Large Entity Relationship Models (PhD Thesis), Melbourne, Australia: Department Of Information Systems, University of Melbourne.]]

[40]

Moody, D. L. and G. G. Shanks (2003): Improving the Quality of Data Models: Empirical Validation of a Quality Management Framework, International Journal of Information Systems, forthcoming.]]

Digital Library

[41]

Morris, C. W. (1970): Foundations of the Theory of Signs, Chicago: Chicago University Press.]]

[42]

Nordbotten, J. C. and M. E. Crosby (1999): The Effect of Graphic Style on Data Model Interpretation, Information Systems Journal, 9.]]

[43]

Nunally, J. (1978): Psychometric Theory (2nd edition), New York: McGraw Hill.]]

[44]

Olle, T. W., H. G. Sol, and A. A. Verrijn-Stuart, eds. Information Systems Design Methodologies: A Comparative Review. 1982, North-Holland: Amsterdam.]]

Digital Library

[45]

Olle, T. W., H. G. Sol, and C. J. Tully, eds. Information Systems Design Methodologies: A Feature Analysis. 1983, North-Holland: Amsterdam.]]

[46]

Olle, T. W., H. G. Sol, and A. A. Verrijn-Stuart, eds. Information Systems Design Methodologies: Improving the Practice. 1986, North-Holland: Amsterdam.]]

[47]

Pedhazur, E. J. and L. P. Schmelkin (1991): Measurement, Design and Analysis: An Integrated Approach, Hillsdale, NJ: Lawrence Erlbaum Associates Inc.]]

[48]

Rescher, N. (1977): Methodological Pragmatism: Systems-Theoretic Approach to the Theory of Knowledge, Oxford: Basil Blackwell.]]

[49]

Rosenthal, R. and D. B. Rubin (1979): A Note on Percent Variance Explained as a Measure of Importance of Effects, Journal of Educational Psychology, 74.]]

[50]

Rosenthal, R. (1983): Assessing the Statistical and Social Importance of the Effects of Psychotherapy, Journal of Consulting and Clinical Psychology, 51.]]

[51]

Shoval, P. and M. Even-Chaime (1987): Database Schema Design: An Experimental Comparison Between Normalisation and Information Analysis, Database, 18 (3), Spring: p. 30--39.]]

Digital Library

[52]

Shoval, P. and S. Shiran (1997): Entity-Relationship and Object-Oriented Data Modeling: An Experimental Comparison of Design Quality, Data & Knowledge Engineering, 21: p. 297--315.]]

Digital Library

[53]

Sølvberg, A. (1999): Data and what they refer to, In Conceptual Modeling, Current Issues and Future Directions (Selected Papers from the Symposium on Conceptual Modeling, Los Angeles, CA, held before ER'97). P. P. Chen, et al. (Eds.), Springer Verlag: Berlin, p. 211--226.]]

Digital Library

[54]

Standish Group (1995): The CHAOS Report into Project Failure, The Standish Group International Inc. Available on-line at http://www.standishgroup.com/visitor/chaos.htm.]]

[55]

Standish Group (1996): Unfinished Voyages, The Standish Group International Inc. available on-line at http://www.standishgroup.com/visitor/voyages.htm.]]

[56]

van Vliet, H. (2000): Software Engineering: Principles and Practice (2nd edition): John Wiley & Sons.]]

Digital Library

[57]

von Halle, B. (1991): Data: Asset or Liability?, Database Programming and Design, 4 (7), July.]]

[58]

Walrad, C. and E. Moss (1993): Measurement: The Key to Application Development Quality, IBM Systems Journal, 32 (3): p. 445--460.]]

Digital Library

[59]

Weber, R. A. (1996): Are Attributes Entities? A Study Of Database Designers' Memory Structures, Information Systems Research, June.]]

[60]

Weber, R. A. (1997): Ontological Foundations Of Information Systems, Melbourne, Australia: Coopers And Lybrand Accounting Research Methodology Monograph No. 4, Coopers And Lybrand.]]

[61]

Westrup, C. (1993): Information Systems Methodologies in Use, Journal of Information Technology, 8.]]

[62]

Witt, G. C. and G. C. Simsion (2000): Data Modeling Essentials: Analysis, Design, and Innovation, The Coriolis Group.]]

Digital Library

[63]

Wynekoop, J. L. and N. L. Russo (1997): Studying Systems Development Methodologies: An Examination Of Research Methods, Information Systems Journal, 7 (1), January.]]

[64]

Zultner, R. E. (1992): The Deming Way: Total Quality Management for Software, In Proceedings of Total Quality Management for Software Conference, Washington, DC, April.]]

Cited By

Mohammadi MMukhtar M(2017)A Service-oriented Methodology for Supporting Business Process Architecture Layers in Supply Chain ManagementInternational Journal of Information Systems and Supply Chain Management10.4018/IJISSCM.201710010210:4(18-43)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJISSCM.2017100102
Izurieta CRojas GGriffith IKruchten POzkaya IKoziolek H(2015)Preemptive Management of Model Driven Technical Debt for Improving Software QualityProceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures10.1145/2737182.2737193(31-36)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1145/2737182.2737193
Chua FChin SHaw S(2014)X-CMInternational Journal of Information Retrieval Research10.4018/ijirr.20141001024:4(17-41)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.4018/ijirr.2014100102
Show More Cited By

Index Terms

Evaluating the quality of information models: empirical testing of a conceptual model quality framework

Recommendations

Are comprehensive quality models necessary for evaluating software quality?
PROMISE '13: Proceedings of the 9th International Conference on Predictive Models in Software Engineering

The concept of software quality is very complex and has many facets. Reflecting all these facets and at the same time measuring everything related to these facets results in comprehensive but large quality models and extensive measurements. In contrast, ...
Evaluating the Quality of Process Models: Empirical Testing of a Quality Framework
ER '02: Proceedings of the 21st International Conference on Conceptual Modeling

This paper conducts an empirical analysis of a conceptual model quality framework for evaluating the quality of process models. 194 participants were trained in the concepts of the quality framework, and then used it to evaluate models represented in a ...
Software quality models: a systematic mapping study
ICSSP '19: Proceedings of the International Conference on Software and System Processes

Quality Models play a critical role in assuring quality and have evolved over 40+ years. They provide support for defining quality attributes, building and measuring the quality of the resulting product. Each quality model adopts a critical view on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '03: Proceedings of the 25th International Conference on Software Engineering

May 2003

841 pages

ISBN:076951877X

General Chair:
Lori Clarke
University of Massachusetts
,
Program Chairs:
Laurie Dillon
Michigan State University
,
Walter Tichy
University of Karlsruhe, Germany

Copyright © Copyright (c) 2003 Institute of Electrical and Electronics Engineers, Inc. All rights reserved.

Sponsors

IEEE Computer Society Technical Council on Software Engineering
SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Computer Society

United States

Publication History

Published: 03 May 2003

Check for updates

Qualifiers

Article

Conference

ICSE03

Sponsor:

SIGSOFT

ICSE03: 25th International Conference on Software Engineering

May 3 - 10, 2003

Oregon, Portland

Acceptance Rates

ICSE '03 Paper Acceptance Rate 42 of 324 submissions, 13%;

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
2,821
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mohammadi MMukhtar M(2017)A Service-oriented Methodology for Supporting Business Process Architecture Layers in Supply Chain ManagementInternational Journal of Information Systems and Supply Chain Management10.4018/IJISSCM.201710010210:4(18-43)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.4018/IJISSCM.2017100102
Izurieta CRojas GGriffith IKruchten POzkaya IKoziolek H(2015)Preemptive Management of Model Driven Technical Debt for Improving Software QualityProceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures10.1145/2737182.2737193(31-36)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1145/2737182.2737193
Chua FChin SHaw S(2014)X-CMInternational Journal of Information Retrieval Research10.4018/ijirr.20141001024:4(17-41)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.4018/ijirr.2014100102
Tiwari NRavikumar GJoshi RJanakiram DSen KKulkarni V(2014)Towards end-to-end multi-dimensional quality evaluation of business processesProceedings of the 7th India Software Engineering Conference10.1145/2590748.2590762(1-6)Online publication date: 19-Feb-2014
https://dl.acm.org/doi/10.1145/2590748.2590762
Le Pallec XDupuy-Chessa S(2013)Support for quality metrics in metamodellingProceedings of the Second Workshop on Graphical Modeling Language Development10.1145/2489820.2489825(23-31)Online publication date: 2-Jul-2013
https://dl.acm.org/doi/10.1145/2489820.2489825
Vicente Pda Silva M(2011)A conceptual model for integrated governance, risk and complianceProceedings of the 23rd international conference on Advanced information systems engineering10.5555/2026716.2026738(199-213)Online publication date: 20-Jun-2011
https://dl.acm.org/doi/10.5555/2026716.2026738
Rockwell SBajaj A(2010)Applying Cognitive Theories to Evaluate Conceptual Models in Systems AnalysisJournal of Information Technology Research10.4018/jitr.20100101053:1(55-72)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.4018/jitr.2010010105
Rittgen PShin SOssowski SSchumacher MPalakal MHung C(2010)Quality and perceived usefulness of process modelsProceedings of the 2010 ACM Symposium on Applied Computing10.1145/1774088.1774105(65-72)Online publication date: 22-Mar-2010
https://dl.acm.org/doi/10.1145/1774088.1774105
Mehmood KSi-Said Cherfi SComyn-Wattiau IMarcos EPapazoglou MPiattini M(2009)Data quality through model qualityProceedings of the first international workshop on Model driven service engineering and data quality and security10.1145/1651415.1651421(29-32)Online publication date: 6-Nov-2009
https://dl.acm.org/doi/10.1145/1651415.1651421
Patig S(2008)A practical guide to testing the understandability of notationsProceedings of the fifth Asia-Pacific conference on Conceptual Modelling - Volume 7910.5555/1379429.1379438(49-58)Online publication date: 1-Jan-2008
https://dl.acm.org/doi/10.5555/1379429.1379438
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents