article

Using mapping studies as the basis for further research - A participant-observer case study

Authors:

Barbara A. Kitchenham,

O. Pearl BreretonAuthors Info & Claims

Information and Software Technology, Volume 53, Issue 6

Pages 638 - 651

https://doi.org/10.1016/j.infsof.2010.12.011

Published: 01 June 2011 Publication History

Abstract

Context: We are strong advocates of evidence-based software engineering (EBSE) in general and systematic literature reviews (SLRs) in particular. We believe it is essential that the SLR methodology is used constructively to support software engineering research. Objective: This study aims to assess the value of mapping studies which are a form of SLR that aims to identify and categorise the available research on a broad software engineering topic. Method: We used a multi-case, participant-observer case study using five examples of studies that were based on preceding mapping studies. We also validated our results by contacting two other researchers who had undertaken studies based on preceding mapping studies and by assessing review comments related to our follow-on studies. Results: Our original case study identified 11 unique benefits that can accrue from basing research on a preceding mapping study of which only two were case specific. We also identified nine problems associated with using preceding mapping studies of which two were case specific. These results were consistent with the information obtained from the validation activities. We did not find an example of an independent research group making use of a mapping study produced by other researchers. Conclusion: Mapping studies can save time and effort for researchers and provide baselines to assist new research efforts. However, they must be of high quality in terms of completeness and rigour if they are to be a reliable basis for follow-on research.

References

[1]

Dyb&#229;, T., Kitchenham, B. and J&#248;rgensen, M., Evidence-based software engineering for practitioners. IEEE Software. v22 i1. 58-65.

Digital Library

[2]

M. J&#248;rgensen, T. Dyb&#229;, B.A. Kitchenham, Teaching evidence-based software engineering to university students, in: 11th IEEE International Software Metrics Symposium (METRICS'05), 2005, p. 24.

Digital Library

[3]

B. Kitchenham, T. Dyb&#229;, M. J&#248;rgensen, Evidence-based software engineering, in: Proceedings of the 26th International Conference on Software Engineering, ICSE '04, IEEE Computer Society, Washington DC, USA, 2004, pp. 273-281.

Digital Library

[4]

Khan, Khalid S., Kunz, Regina, Kleijnen, Jos and Antes, Gerd, Systematic Reviews to Support Evidence-Based Medicine. 2003. The Royal Society of Medicine Press Ltd.

[5]

B.A. Kitchenham, S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering Technical Report EBSE-2007-01, 2007.

[6]

Petticrew, M. and Roberts, H., Systematic Reviews in the Social Sciences: A Practical Guide. 2005. Blackwell Publishing.

[7]

Budgen, D., Bailey, J., Turner, M., Kitchenham, B., Brereton, P. and Charters, S., A cross-domain investigation of empirical practices. IET Software. v3 i5. 410-421.

[8]

O.P. Brereton, B.A. Kitchenham, The Scope of EPIC Case Studies, EPIC Technical Report EPIC-2007-01, 2009.

[9]

D. Budgen, M. Turner, P. Brereton, B. Kitchenham, Using mapping studies in software engineering, in: Proceedings of PPIG 2008, Lancaster University, 2008, pp. 195-204.

[10]

Kitchenham, B., Pretorius, R., Budgen, D., Brereton, P., Turner, M., Niazi, M. and Linkman, S.G., Systematic literature reviews in software engineering - a tertiary study. Information and Software Technology. v52 i8. 934-944.

Digital Library

[11]

B. Kitchenham, D. Budgen, P. Brereton, The value of mapping studies - a participant observer case study, in: Proceedings of EASE 2010, BCS eWic, 2010.

Digital Library

[12]

J&#248;rgensen, M., A review of studies on expert estimation of software development effort. Journal of Systems and Software. v70 i1-2. 37-60.

Digital Library

[13]

K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, Systematic mapping studies in software engineering, in: Proceedings EASE 08, BCS eWIC, 2008.

Digital Library

[14]

Kitchenham, B.A., What's up with metrics? - a preliminary mapping study. Journal of Systems and Software. v83 i1. 37-51.

Digital Library

[15]

D. Budgen, A.J.Burn, O.P. Brereton, B.A. Kitchenham, R. Pretorius, Empirical evidence about the UML: a systematic literature review, Software: practice and experience, in press.

Digital Library

[16]

J&#248;rgensen, M. and Shepperd, M., A systematic review of software development cost estimation studies. IEEE Transactions on Software Engineering. v33 i1. 33-53.

Digital Library

[17]

Sj&#248;berg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N.K. and Rekdal, A.C., A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering. v31 i9. 733-753.

Digital Library

[18]

H. Zhang, B. Kitchenham, D. Pfahl, Software process simulation modeling: facts, trends, and directions, in: Proceedings of 15th Asia-Pacific Software Engineering Conference (APSEC'08), Beijing, China, IEEE Computer Society, 2008, pp. 59-66.

Digital Library

[19]

Dyb&#229;, T., Kampenes, V.B. and Sj&#248;berg, D.I.K., A systematic review of statistical power in software engineering experiments. Information and Software Technology. v48 i8. 745-755.

[20]

Hannay, J. and J&#248;rgensen, M., The role of deliberate artificial design elements in software engineering experiments. IEEE Transactions on Software Engineering. v34 i2. 242-259.

Digital Library

[21]

Hannay, J.E., Sj&#248;berg, D.I.K. and Dyb&#229;, T., A systematic review of theory use in software engineering experiments. IEEE Transactions on Software Engineering. v33 i2. 87-107.

Digital Library

[22]

Kampenes, V.B., Dyb&#229;, T., Hannay, J.E. and Sj&#248;berg, Dag I.K., A systematic review of effect size in software engineering experiments. Information and Software Technology. v49 i11-12. 1073-1086.

Digital Library

[23]

Kampenes, V.B., Dyb&#229;, T., Hannay, J.E. and Sj&#248;berg, DagI.K., A systematic review of quasi-experiments in software engineering. Information and Software Technology. v51 i1. 71-82.

Digital Library

[24]

J. Bailey, D. Budgen, M. Turner, B. Kitchenham, P. Brereton, S. Linkman, Evidence relating to object-oriented software design: a survey, in: Proceedings of Empirical Software Engineering and Measurement, 2007, IEEE Computer Society Press, 2007, pp. 482-484.

Digital Library

[25]

G&#243;mez, O., Oktaba, H., Piattini, M. and Garc&#237;***a, F., A systematic review measurement in software engineering: state-of-the-art in measures, ICSOFT 2006. CCIS. v10. 165-176.

[26]

Bellini, C.G., Pereira, R.D.C.D.F. and Becker, J.L., Measurement in software engineering from the roadmap to the crossroads. International Journal of Software Engineering and Knowledge Engineering. v18 i1. 37-64.

[27]

Catal, C. and Diri, B., A systematic review of software fault prediction studies. Expert Systems with Applications. v36. 7346-7354.

Digital Library

[28]

Kitchenham, B., Brereton, O.P., Budgen, D., Turner, M., Bailey, J. and Linkman, S., Systematic literature reviews in software engineering - a systematic literature review. Information and Software Technology. v51. 7-15.

Digital Library

[29]

Neto, A.D., Subramanyan, R., Viera, M., Travassos, G.H. and Shull, F., Improving evidence about software technologies: a look at model-based testing. IEEE Software. v25 i6. 242-249.

Digital Library

[30]

Kitchenham, B.A., Mendes, E. and Travassos, G.H., Cross versus within-company cost estimation studies: a systematic review. IEEE Transactions on Software Engineering. v33 i5. 316-329.

Digital Library

[31]

B. Kitchenham, P. Brereton, D. Budgen, The Educational Value of Mapping Studies of Software Engineering Literature, ICSE 2010 Education Theme, ACM Press, South Africa, 2010, pp. 1-7.

Digital Library

[32]

Wieringa, R., Maiden, N.A.M., Mead, N.R. and Rolland, C., Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements Engineering. v11 i1. 102-107.

Digital Library

[33]

R. Pretorius, D. Budgen, A mapping study on empirical evidence related to the models and forms used in the UML, in: Proceedings of 2nd ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2008, ACM Press, 2008, pp. 342-344.

Digital Library

[34]

C. Zhang, D. Budgen, What Do We Know about the Effectiveness of Software Design Patterns, submitted for publication.

[35]

Yin, Robert K., Case Study Research: Design and Methods. 2003. third ed. Sage Publications.

[36]

B. Kitchenham, P. Brereton, M. Turner, M. Niazi, S.G. Linkman, R. Pretorius, D. Budgen, The impact of limited search procedures for systematic literature reviews - an observer-participant case study, in: Proceedings of ESEM 2009, IEEE CS Press, 2009.

Digital Library

[37]

D. Budgen, C. Zhang, Preliminary reporting guidelines for experience papers, in: Proceedings of EASE 2009, BCS eWiC, 2009.

Digital Library

[38]

Kitchenham, B., Brereton, P., Turner, M., Niazi, M., Linkman, S.G., Pretorius, R. and Budgen, D., Refining the systematic literature review process - two participant observer case studies. Empirical Software Engineering. v15 i6. 618-653.

Digital Library

[39]

H. Zhang, B. Kitchenham, D. Pfahl, Reflections on 10 years of software process simulation modelling: a systematic review, in: Proceedings of International Conference on Software Process (ICSP'08), vol. LNCS 5007, Springer-Verlag, Leipzig, Germany, 2008, pp. 345-365.

Digital Library

[40]

H. Zhang, B. Kitchenham, D. Phahl, Software process simulation modeling: an extended systematic review, in: International Conference on Software Process ICSP, 2010.

Digital Library

[41]

Afzal, W., Torker, R. and Feldt, R., A systematic review of search-based testing for non-functional system properties. Information and Software Technology. v51 i6. 957-976.

Digital Library

[42]

W. Afzal, R. Torker, R. Feldt, A systematic mapping study on non-functional search-based software testing, in: 20th International Conference on Software Engineering and Knowledge Engineering (SEKE), 2008.

[43]

S. Beecham, A Short Comparison of Two Systematic Literature Reviews on Fault Prediction, University of Hertfordshire Technical Report UH-CS-TR-492, 2009.

Cited By

Li TZhang XWang YZhou QWang YDong F(2024)Machine learning for requirements engineering (ML4RE)Information and Software Technology10.1016/j.infsof.2024.107477172:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107477
Díaz OGenero MContell JPiattini M(2024)Adding relevance to rigorComputer Science Review10.1016/j.cosrev.2024.10064953:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.cosrev.2024.100649
Caraveo-Cacep MVázquez-Medina RHernández Zavala A(2024)A review on security implementations in soft-processors for IoT applicationsComputers and Security10.1016/j.cose.2023.103677139:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.cose.2023.103677
Show More Cited By

Recommendations

The value of mapping studies: a participantobserver case study
EASE'10: Proceedings of the 14th international conference on Evaluation and Assessment in Software Engineering

We are strong advocates of evidence-based software engineering (EBSE) in general and systematic literature reviews (SLRs) in particular. We believe it is essential that the SLR methodology is being used constructively to support software engineering ...
Mapping the systematic literature studies about software ecosystems
TEEM'18: Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality

There is a need to improve the definition and development of technological ecosystems in order to solve the main problems detected in previous studies. To achieve this goal, it is required to identify and analyse the solutions available in the ...
Refining the systematic literature review process--two participant-observer case studies

Systematic literature reviews (SLRs) are a major tool for supporting evidence-based software engineering. Adapting the procedures involved in such a review to meet the needs of software engineering and its literature remains an ongoing process. As part ...

Comments

Information & Contributors

Information

Published In

cover image Information and Software Technology

Information and Software Technology Volume 53, Issue 6

June, 2011

135 pages

ISSN:0950-5849

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2010.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 01 June 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

132
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li TZhang XWang YZhou QWang YDong F(2024)Machine learning for requirements engineering (ML4RE)Information and Software Technology10.1016/j.infsof.2024.107477172:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107477
Díaz OGenero MContell JPiattini M(2024)Adding relevance to rigorComputer Science Review10.1016/j.cosrev.2024.10064953:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.cosrev.2024.100649
Caraveo-Cacep MVázquez-Medina RHernández Zavala A(2024)A review on security implementations in soft-processors for IoT applicationsComputers and Security10.1016/j.cose.2023.103677139:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.cose.2023.103677
Nooribakhsh MFernández-Diego MGonzález-Ladrón-De-Guevara FMollamotalebi M(2024)Community detection in social networks using machine learning: a systematic mapping studyKnowledge and Information Systems10.1007/s10115-024-02201-866:12(7205-7259)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10115-024-02201-8
Castro JMadrigal GRojas L(2024)Eye Tracking to Evaluate Usability with the Older Adults: A Secondary StudySocial Computing and Social Media10.1007/978-3-031-61281-7_9(131-145)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-61281-7_9
de Campos VDavid JStröele VBraga R(2024)Aligning technical knowledge to an industry domain in global software developmentJournal of Software: Evolution and Process10.1002/smr.271336:12Online publication date: 10-Dec-2024
https://dl.acm.org/doi/10.1002/smr.2713
Carbonera CFarias KGraeff CSilva R(2023)Software Merge: A Two-Decade Systematic Mapping StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613391(99-108)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613391
Vainio-Pekka HAgbese MJantunen MVakkuri VMikkonen TRousi RAbrahamsson P(2023)The Role of Explainable AI in the Research Field of AI EthicsACM Transactions on Interactive Intelligent Systems10.1145/359997413:4(1-39)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1145/3599974
Kotti ZGalanopoulou RSpinellis D(2023)Machine Learning for Software Engineering: A Tertiary StudyACM Computing Surveys10.1145/357290555:12(1-39)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3572905
Kokkoniemi MIsomöttönen V(2023)A systematic mapping study on group work research in computing education projectsJournal of Systems and Software10.1016/j.jss.2023.111795204:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111795
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents