research-article

Open access

Multivocal Literature Review on DevOps Critical Success Factors

Authors:

Sami HyrynsalmiAuthors Info & Claims

EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

Pages 520 - 527

https://doi.org/10.1145/3661167.3661236

Published: 18 June 2024 Publication History

All formats PDF

Abstract

DevOps is a methodology that seeks to unify development and operations teams in organizations, aiming to facilitate faster software delivery and promote collaboration to build a positive company culture. Our research aims to investigate the current state-of-the-art of DevOps, align academic research with industry practices, and identify critical success factors. We conducted a comprehensive literature review using a variety of databases and search engines, which revealed that several factors are essential to the success of DevOps, including DevOps culture, automation processes, continuous integration, and deployment, monitoring, and feedback, standardization with tools, team leadership, and DecSecOps for security issues. While DevOps has gained significant attention, it remains essential to understand practitioners’ perspectives. Our research has the potential to strengthen the concepts and ideas of critical success factors, broaden DevOps practices and perspectives for professionals, and enhance academic knowledge in this area.

References

[1]

Pekka Abrahamsson, Outi Salo, Jussi Ronkainen, and Juhani Warsta. 2017. Agile software development methods: Review and analysis. arXiv preprint arXiv:1709.08439 (2017).

[2]

Muhammad Azeem Akbar, Sajjad Mahmood, Muhammad Shafiq, Ahmed Alsanad, Abeer Abdul-Aziz Alsanad, and Abdu Gumaei. 2020. Identification and prioritization of DevOps success factors using fuzzy-AHP approach. Soft Computing (2020), 1–25.

[3]

Betool Alnamlah, Sara Alshathry, Norah Alkassim, and Nor Jamail. 2021. The necessity of a lead person to monitor development stages of the DevOps pipeline. Indonesian Journal of Electrical Engineering and Computer Science 21 (01 2021), 348. https://doi.org/10.11591/ijeecs.v21.i1.pp348-353

[4]

Ricardo Manuel Duarte Amaro, Ruben Pereira, and Miguel Mira da Silva. 2022. Capabilities and practices in DevOps: a multivocal literature review. IEEE Transactions on Software Engineering (2022).

[5]

Nasreen Azad. 2022. Understanding DevOps critical success factors and organizational practices. In 2022 IEEE/ACM International Workshop on Software-Intensive Business (IWSiB). IEEE, 83–90.

Digital Library

[6]

Nasreen Azad and Sami Hyrynsalmi. 2021. What Are Critical Success Factors of DevOps Projects? A Systematic Literature Review. In Software Business(Lecture Notes in Business Information Processing, Vol. 434), Xiaofeng Wang, Antonio Martini, Anh Nguyen-Duc, and Viktoria Stray (Eds.). Springer, Cham. https://doi.org/10.1007/978-3-030-91983-2_17

[7]

Nasreen Azad and Sami Hyrynsalmi. 2022. DevOps Challenges in Organizations: Through Professional Lens. In Software Business: 13th International Conference, ICSOB 2022, Bolzano, Italy, November 8–11, 2022, Proceedings. Springer, 260–277.

[8]

Nasreen Azad and Sami Hyrynsalmi. 2023. DevOps critical succes factors—A systematic literature review. Information and Software Technology (2023), 107150.

[9]

Nasreen Azad and Sami Hyrynsalmi. 2023. DevOps critical succes factors—A systematic literature review. Information and Software Technology (2023), 107150.

[10]

C Banerjee and SK Pandey. 2009. Software security rules, SDLC perspective. arXiv preprint arXiv:0911.0494 (2009).

[11]

Walid Ben Mesmia, Mohamed Escheikh, and Kamel Barkaoui. 2021. DevOps workflow verification and duration prediction using non-Markovian stochastic Petri nets. Journal of Software: Evolution and Process 33, 3 (2021), e2329.

Digital Library

[12]

Cor-Paul Bezemer, Simon Eismann, Vincenzo Ferme, Johannes Grohmann, Robert Heinrich, Pooyan Jamshidi, Weiyi Shang, André van Hoorn, Monica Villavicencio, Jürgen Walter, 2019. How is performance addressed in DevOps?. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. 45–50.

Digital Library

[13]

Moroz Bogdan, Andrey Saltan, and Sami Hyrynsalmi. 2023. Product Ops: Understanding and Defining an Emerging Discipline. In Int.Conf. on Software Product Management. GI, 11.

[14]

Lianping Chen. 2017. Continuous delivery: overcoming adoption challenges. Journal of Systems and Software 128 (2017), 72–86.

Digital Library

[15]

Gerry Gerard Claps, Richard Berntsson Svensson, and Aybüke Aurum. 2015. On the journey to continuous deployment: Technical and social challenges along the way. Information and Software technology 57 (2015), 21–31.

[16]

Patrick Debois. 2008. Agile infrastructure and operations: how infra-gile are you?. In Agile 2008 Conference. IEEE, 202–207.

Digital Library

[17]

Jessica Díaz, Daniel López-Fernández, Jorge Pérez, and Ángel González-Prieto. 2021. Why are many businesses instilling a DevOps culture into their organization?Empirical Software Engineering 26, 2 (2021), 1–50.

[18]

Jessica Díaz, Jorge E Perez, Agustin Yague, Andrea Villegas, and Antonio de Antona. 2019. Devops in practice–a preliminary analysis of two multinational companies. In International Conference on Product-Focused Software Process Improvement. Springer, 323–330.

[19]

Tore Dyba and Torgeir Dingsoyr. 2009. What do we know about agile software development?IEEE software 26, 5 (2009), 6–9.

[20]

Andrej Dyck, Ralf Penners, and Horst Lichter. 2015. Towards definitions for release engineering and DevOps. In 2015 IEEE/ACM 3rd International Workshop on Release Engineering. IEEE, 3–3.

Digital Library

[21]

Floris MA Erich, Chintan Amrit, and Maya Daneva. 2017. A qualitative study of DevOps usage in practice. Journal of software: Evolution and Process 29, 6 (2017), e1885.

Digital Library

[22]

Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. 2013. Development and deployment at facebook. IEEE Internet Computing 17, 4 (2013), 8–17.

Digital Library

[23]

Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123 (2017), 176–189.

[24]

Nicole Forsgren and Jez Humble. 2016. The role of continuous delivery in IT and organizational performance. Forsgren, N., J. Humble (2016)." The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI) (2016).

[25]

Martin Fowler and Matthew Foemmel. 2006. Continuous integration.

[26]

Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology 106 (2019), 101–121.

[27]

Rogelio Hernández, Begoña Moros, and Joaquín Nicolás. 2023. Requirements management in DevOps environments: a multivocal mapping study. Requirements Engineering (2023), 1–30.

[28]

Jez Humble and David Farley. 2010. Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education.

[29]

Jez Humble and Gene Kim. 2018. Accelerate: The science of lean software and devops: Building and scaling high performing technology organizations. IT Revolution.

[30]

Jon Iden, Bjørnar Tessem, and Tero Päivärinta. 2011. Problems in the interplay of development and IT operations in system development projects: A Delphi study of Norwegian IT experts. Information and Software Technology 53, 4 (2011), 394–406.

Digital Library

[31]

Barbara A Kitchenham. 2012. Systematic review in software engineering: where we are and where we should be going. In Proceedings of the 2nd international workshop on Evidential assessment of software technologies. 1–2.

Digital Library

[32]

Gwendolyn L Kolfschoten, Gert-Jan de Vreede, Robert O Briggs, and Henk G Sol. 2010. Collaboration ‘engineerability’. Group Decision and Negotiation 19, 3 (2010), 301–321.

[33]

Sara Kraemer, Pascale Carayon, and John Clem. 2009. Human and organizational factors in computer and information security: Pathways to vulnerabilities. Computers & security 28, 7 (2009), 509–520.

[34]

JANE LAUDON LAUDON. 2017. PEARSON MYLAB MIS WITH PEARSON ETEXT-INSTANTACCESS-MANAGEMENT INFORMATION SYSTEMS:... Managing the Digital Firm, Global Edition. PEARSON EDUCATION LIMITED.

[35]

Eero Laukkanen, Juha Itkonen, and Casper Lassenius. 2017. Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology 82 (2017), 55–79.

[36]

Timo OA Lehtinen, Mika V Mäntylä, Jari Vanhanen, Juha Itkonen, and Casper Lassenius. 2014. Perceived causes of software project failures–An analysis of their relationships. Information and Software Technology 56, 6 (2014), 623–643.

Digital Library

[37]

Joel K Leidecker and Albert V Bruno. 1984. Identifying and using critical success factors. Long range planning 17, 1 (1984), 23–32.

[38]

Iraj Lohrasbinasab, Prameet Bhakta Acharya, and Ricardo Colomo-Palacios. 2020. BizDevOps: a multivocal literature review. In Computational Science and Its Applications–ICCSA 2020: 20th International Conference, Cagliari, Italy, July 1–4, 2020, Proceedings, Part VI 20. Springer, 698–713.

[39]

Welder Pinheiro Luz, Gustavo Pinto, and Rodrigo Bonifácio. 2018. Building a collaborative culture: a grounded theory of well succeeded devops adoption in practice. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10.

Digital Library

[40]

Lucy Ellen Lwakatare, Teemu Karvonen, Tanja Sauvola, Pasi Kuvaja, Helena Holmström Olsson, Jan Bosch, and Markku Oivo. 2016. Towards DevOps in the embedded systems domain: Why is it so hard?. In 2016 49th hawaii international conference on system sciences (hicss). IEEE, 5437–5446.

[41]

Ruth W Macarthy and Julian M Bass. 2020. An empirical taxonomy of DevOps in practice. In 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 221–228.

[42]

Simo Mäkinen, Marko Leppänen, Terhi Kilamo, Anna-Liisa Mattila, Eero Laukkanen, Max Pagels, and Tomi Männistö. 2016. Improving the delivery cycle: A multiple-case study of the toolchains in Finnish software intensive enterprises. Information and Software Technology 80 (2016), 175–194.

Digital Library

[43]

Runfeng Mao, He Zhang, Qiming Dai, Huang Huang, Guoping Rong, Haifeng Shen, Lianping Chen, and Kaixiang Lu. 2020. Preliminary findings about devsecops from grey literature. In 2020 IEEE 20th international conference on software quality, reliability and security (QRS). IEEE, 450–457.

[44]

Gary McGraw. 2004. Software security. IEEE Security & Privacy 2, 2 (2004), 80–83.

Digital Library

[45]

Nils Brede Moe, Torgeir Dingsøyr, and Tore Dybå. 2008. Understanding self-organizing teams in agile software development. In 19th australian conference on software engineering (aswec 2008). IEEE, 76–85.

Digital Library

[46]

Vaishnavi Mohan and Lotfi Ben Othmane. 2016. Secdevops: Is it a marketing buzzword?-mapping research on security in devops. In 2016 11th international conference on availability, reliability and security (ARES). IEEE, 542–547.

[47]

Haavard Myrbakken and Ricardo Colomo-Palacios. 2017. DevSecOps: a multivocal literature review. In Software Process Improvement and Capability Determination: 17th International Conference, SPICE 2017, Palma de Mallorca, Spain, October 4–5, 2017, Proceedings. Springer, 17–29.

[48]

Kristian Nybom, Jens Smeds, and Ivan Porres. 2016. On the impact of mixing responsibilities between devs and ops. In International Conference on Agile Software Development. Springer, 131–143.

[49]

Marta Olszewska and Marina Waldén. 2015. DevOps meets formal modelling in high-criticality complex systems. In Proceedings of the 1st international workshop on quality-aware DevOps. 7–12.

Digital Library

[50]

Kai Petersen and Claes Wohlin. 2009. A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. Journal of systems and software 82, 9 (2009), 1479–1490.

Digital Library

[51]

Minna Pikkarainen, Jukka Haikara, Outi Salo, Pekka Abrahamsson, and Jari Still. 2008. The impact of agile practices on communication in software development. Empirical Software Engineering 13 (2008), 303–337.

Digital Library

[52]

Luís Prates, João Faustino, Miguel Silva, and Rúben Pereira. 2019. Devsecops metrics. In Information Systems: Research, Development, Applications, Education: 12th SIGSAND/PLAIS EuroSymposium 2019, Gdansk, Poland, September 19, 2019, Proceedings 12. Springer, 77–90.

[53]

Leah Riungu-Kalliosaari, Simo Mäkinen, Lucy Ellen Lwakatare, Juha Tiihonen, and Tomi Männistö. 2016. DevOps adoption benefits and challenges in practice: a case study. In International conference on product-focused software process improvement. Springer, 590–597.

[54]

John F Rockart. 1979. Chief executives define their own data needs.Harvard business review 57, 2 (1979), 81–93.

[55]

Pilar Rodríguez, Alireza Haghighatkhah, Lucy Ellen Lwakatare, Susanna Teppola, Tanja Suomalainen, Juho Eskeli, Teemu Karvonen, Pasi Kuvaja, June M Verner, and Markku Oivo. 2017. Continuous deployment of software intensive products and services: A systematic mapping study. Journal of systems and software 123 (2017), 263–291.

[56]

Mary Sánchez-Gordón and Ricardo Colomo-Palacios. 2018. A multivocal literature review on the use of DevOps for e-learning systems. In Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality. 883–888.

Digital Library

[57]

Mojtaba Shahin. 2015. Architecting for devops and continuous deployment. In Proceedings of the ASWEC 2015 24th Australasian Software Engineering Conference. 147–148.

Digital Library

[58]

Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE access 5 (2017), 3909–3943.

[59]

Jens Smeds, Kristian Nybom, and Ivan Porres. 2015. DevOps: a definition and perceived adoption impediments. In Agile Processes in Software Engineering and Extreme Programming: 16th International Conference, XP 2015, Helsinki, Finland, May 25-29, 2015, Proceedings 16. Springer, 166–177.

[60]

Darja Šmite, Nils Brede Moe, and Javier Gonzalez-Huerta. 2021. Overcoming cultural barriers to being agile in distributed teams. Information and Software Technology 138 (2021), 106612.

[61]

Daniel Ståhl and Jan Bosch. 2014. Automated software integration flows in industry: A multiple-case study. In Companion Proceedings of the 36th International Conference on Software Engineering. 54–63.

Digital Library

[62]

Michiel Van Belzen, Jos Trienekens, and Rob Kusters. 2019. Critical success factors of continuous practices in a DevOps context. In 28th International Conference of Information Systems Development (ISD 2019).

[63]

Johannes Wettinger, Uwe Breitenbücher, and Frank Leymann. 2014. Devopslang–bridging the gap between development and operations. In European Conference on Service-Oriented and Cloud Computing. Springer, 108–122.

[64]

Anna Wiedemann, Manuel Wiesche, Heiko Gewald, and Helmut Krcmar. 2020. Understanding how DevOps aligns development and operations: a tripartite model of intra-IT alignment. European Journal of Information Systems 29, 5 (2020), 458–473.

[65]

Changbyung Yoon, Keeeun Lee, Byungun Yoon, and Omar Toulan. 2017. Typology and success factors of collaboration for sustainable growth in the IT service industry. Sustainability 9, 11 (2017).

[66]

Liming Zhu, Len Bass, and George Champlin-Scharff. 2016. DevOps and its practices. IEEE software 33, 3 (2016), 32–34.

Cited By

Pahune SAkhtar Z(2025)Transitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language ModelsInformation10.3390/info1602008716:2(87)Online publication date: 23-Jan-2025
https://doi.org/10.3390/info16020087
Azad NHyrynsalmi S(2024)Navigating Challenges and Pitfalls in DevOps-Driven Digital TransformationDisruptive Innovation in a Digitally Connected Healthy World10.1007/978-3-031-72234-9_10(115-128)Online publication date: 10-Sep-2024
https://doi.org/10.1007/978-3-031-72234-9_10

Recommendations

Understanding DevOps critical success factors and organizational practices
IWSiB '22: Proceedings of the 5th International Workshop on Software-intensive Business: Towards Sustainable Software Business

DevOps is a combination of practices and a company culture which aims minimize the barriers between the operation and development teams in the organization. As its adoption and use in the industry has been growing, different kinds of research are trying ...
Understanding DevOps Critical Success Factors: Insights from Professionals
New Sustainable Horizons in Artificial Intelligence and Digital Solutions
Abstract
This paper explores the factors that contribute to the success of software development undertaken with DevOps practices. DevOps is a set of practices that aims to increase software development process efficiency by reducing barriers between ...
DevOps critical success factors — A systematic literature review
Abstract Context:
DevOps is a set of software development and operation practices and a recent addition to a large family of different kinds of software process models. The model emerged out of the observation that information ...
Highlights
- A systematic literature review was conducted to identify critical success factors.

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '24: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering

June 2024

728 pages

ISBN:9798400717017

DOI:10.1145/3661167

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2024

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

EASE 2024

EASE 2024: 28th International Conference on Evaluation and Assessment in Software Engineering

June 18 - 21, 2024

Salerno, Italy

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
290
Total Downloads

Downloads (Last 12 months)290
Downloads (Last 6 weeks)50

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pahune SAkhtar Z(2025)Transitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language ModelsInformation10.3390/info1602008716:2(87)Online publication date: 23-Jan-2025
https://doi.org/10.3390/info16020087
Azad NHyrynsalmi S(2024)Navigating Challenges and Pitfalls in DevOps-Driven Digital TransformationDisruptive Innovation in a Digitally Connected Healthy World10.1007/978-3-031-72234-9_10(115-128)Online publication date: 10-Sep-2024
https://doi.org/10.1007/978-3-031-72234-9_10

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten