Multivocal Literature Review on DevOps Critical Success Factors

Nasreen Azad, Software Engineering, LUT University, Finland, [email protected]
Sami Hyrynsalmi, Department Software Engineering, LUT University, Finland, [email protected]

DOI: https://doi.org/10.1145/3661167.3661236
EASE 2024: 28th International Conference on Evaluation and Assessment in Software Engineering, Salerno, Italy, June 2024

DevOps is a methodology that seeks to unify development and operations teams in organizations, aiming to facilitate faster software delivery and promote collaboration to build a positive company culture. Our research aims to investigate the current state-of-the-art of DevOps, align academic research with industry practices, and identify critical success factors. We conducted a comprehensive literature review using a variety of databases and search engines, which revealed that several factors are essential to the success of DevOps, including DevOps culture, automation processes, continuous integration, and deployment, monitoring, and feedback, standardization with tools, team leadership, and DecSecOps for security issues. While DevOps has gained significant attention, it remains essential to understand practitioners’ perspectives. Our research has the potential to strengthen the concepts and ideas of critical success factors, broaden DevOps practices and perspectives for professionals, and enhance academic knowledge in this area.

CCS Concepts: • Do Not Use This Code, Generate the Correct Terms for Your Paper; • Do Not Use This Code, Generate the Correct Terms for Your Paper; • Do Not Use This Code, Generate the Correct Terms for Your Paper; • Do Not Use This Code, Generate the Correct Terms for Your Paper;

Keywords: Critical Success Factors, DevOps, Multivocal Literature Review, DevOps practices, Software Development

ACM Reference Format:
Nasreen Azad and Sami Hyrynsalmi. 2024. Multivocal Literature Review on DevOps Critical Success Factors. In 28th International Conference on Evaluation and Assessment in Software Engineering (EASE 2024), June 18--21, 2024, Salerno, Italy. ACM, New York, NY, USA 8 Pages. https://doi.org/10.1145/3661167.3661236

1 INTRODUCTION

The business environment of today's world is changing rapidly. To keep pace with this dynamic environment, it is essential to grow with the flow. This change also impacts the software development process. Customer demands and expectations are higher, and they expect the software applications to be provided with higher quality with a quicker delivery time [66]. Everything changed gradually as the technology improved for build initiation, tests, and deployments. The system has become faster and automated and processes everything in a short time [66].

IT organizations are constantly facing challenges to cope with customer requirements and needs. There is competition in the market, and outside threats are another challenge for any company [4, 24]. There could always be a software crisis, and the market cannot provide it timely. Software crisis can be exaggerated, and it is not uncommon for a project to fail completely [36, 38]. It is always a great approach to be ahead by establishing a competitive advantage by working efficiently and faster than competitors in the competitive market [4, 34].

According to Dyck et al.,  [20], it is essential that delivering and supporting software should have reliability. There is always a need for frequent software delivery in the market. If the software is without sustained builds, then a lack of proper testing and release automation might generate failure and create a difficult situation for engineers in operating the process. This might delay the software delivery process and impact reliability [4, 24]. Thus, the DevOps automation process came to light as a part of the revolutionary changes from the past software development practices by IT organizations [12].

In the Software Engineering industry, the entry of DevOps practices has considerably changed the software release process. It has become a new mindset of work practice for development teams and operations teams’ collaboration perspective [41]. To understand the organizational practices, a multi-vocal literature review could help to understand practitioners’ perspectives regarding DevOps practices and align the thoughts of academics and professionals on a similar page. Some multivocal literature review research has been done, for instance, on DevOps usage and practices [21], use of DevOps for e-learning systems [56], characterizing DevOps practices [24], DevOps metrics [52], DecSecOps [47], capabilities and practices in DevOps [4], BizDevOPs [38], ProductOps [13] etc.

In this paper, we have conducted a multivocal literature review followed by the approaches presented by Garousi et al.  [26]. Thus, a multivocal literature review can be used to determine how critical success impacts DevOps's performance and practices in IT organizations. As critical success factors are changing based on the implementation and adoption of DevOps, the company, why Critical success factors are essential for DevOps practices and adoption, and what could be the relationship between the Critical success factors and desired practices of DevOps are still not clear for the DevOps practitioners and DevOps community. This is one of the significant gaps this paper will be addressing.

The remainder of the paper is presented as follows. Section 2 goes through the background: concepts of DevOps and critical success factors. The research process used is explained in Section 3, including prior multivocal literature studies on the topic, research questions, and motivation. In Section 4, the results are reported based on the research questions. Section 5 includes a discussion of threats to validity and future work, and, finally, the last section summarizes the conclusion.

2 BACKGROUND

2.1 DevOps Concepts

DevOps is a software development practice that combines the operations and development teams [5, 6, 8]. The reason that DevOps eliminates the information silos that are faced by the engineering teams while collaborating with teams is based on a specific cultural mindset suggested by [16, 42]. To get faster and more frequent delivery of software, product automation and collaboration across the department is necessary [6]. According to Olszewska et al. a set of principles and practices that accelerates more excellent communication and coordination among key stakeholders for achieving goals for software products and services [26, 49].

There were inefficiencies in the software development process for a long time. The release time was slower, and the operations processes were interrupted by organizational splits between the development process. The team's communication was hampered due to various gaps within the teams. DevOps practices came along to tackle this inefficiency in the software development process. The DevOps concept was first pioneered by a practitioner named Petrik Debois [16, 40]. According to Azad and Hyrynsalmi, DevOps is an infinite loop of operations that follow a sequence that combines seven phases, including planning, code development, building, deploying, testing, operating, and monitoring [8]. DevOps aims to create a common practice where the Development and operations teams work together to achieve an efficient software development process with faster release time [7, 40]. It is common for the development and operations teams to face conflicting issues due to agility and stability concerns. There are also many challenges including insufficient flow of information, knowledge gap, and unstable test environments while working in teams [4, 10, 11, 12, 30, 37]. Below we discuss continuous integration, continuous delivery, and continuous deployment which are considered as one of the three main components of DevOps practices.

2.1.1 Continuous Integration. Continuous integration is a process that connects steps related to code compilations, running unit tests, coverage for code validation, checking code standards, and building packages for deployment [23]. The integration process starts automatically, helps ensure the integration frequency, and handles the highly visible artifacts. Continuous integration solves integration failures faster and maintains the ongoing automated integration flows [61]. According to Fitzgerald, the developers must get better feedback for the integration process, and due to that, the integration frequency should be regular [23].

For the developers, depositing the codes daily in the standard code repository is essential to integrate the work process with continuous integration [25, 62]. The system must be tested and built to check whether the system functions properly to new changes so that other developers can work with the new changes after every integration process [36, 62]. There is an iteration in each integration process that meets a feedback loop. The feedback loop helps to solve any problems related to source code testing. Thus, source changes increase successful implementation and progress through the system [61].

2.1.2 Continuous Delivery. Continuous delivery is an approach used in software engineering that combines a set of principles, practices, and patterns that make the deployment at any time of the development process [14, 29, 62]. For teams, it is essential that software should be produced continuously within a short life cycle so that the decision of release can be made at any time [14, 23, 28]. According to Fitzgerald and Ståhl, continuous delivery increases the software's ability to be deployed in the working environment [23]. According to Chen, continuous delivery has some essential characteristics, which include short cycles, resealable at any time, and release reliability [14]. Fast feedback loops, support for automation, and reliable and quicker releases to the customer are also essential facts to consider for continuous delivery [23, 58, 62].

2.1.3 Continuous Deployment. Continuous deployment helps the codes to be deployed immediately to the production environment so that the customers can use the final outcome of the application [15, 28, 55]. Manual steps or developer commit decisions for the production deployment are needed [35]. This process helps get better user feedback and detect defects, reducing costs. Thus, feedback is acquired faster for the deployment process [35]. For the continuous deployment of agile processes at team levels, integration for research and development and process automation is required [55].

2.2 Critical success factors

Critical success factors, a cornerstone in software engineering, encompass characteristics, conditions, and variables that, when properly managed, maintained, and sustained, can significantly influence the success of the organization [5, 6, 8, 37, 62]. Rockart's critical success factors findings posits that these factors ensure competitive performance by delivering satisfactory results in a limited number of areas within the organization [54]. According to Shahin et al., a factor can be deemed critical if it impacts continuous practices and contributes to the process's success [57]. Azad and Hyrynsalmi further emphasize the importance of understanding various success factors for the company's progress and future prediction of uncertainty [5, 6, 7, 8].

3 RESEARCH PROCESS

3.1 Multivocal literature review

According to Garousi et al.  [26], a multivocal literature review (MLR) is a form of systematic literature review (SLR) that combines some forms of videos, web pages, or blogs that the practitioners produce. Those items do not belong to the academic areas of research as those online contents are not peer-reviewed nor follow academic writing patterns like journals or conference papers, which gradually follow for explaining outcomes of the research [26, 31]. Multivocal literature is a valuable technique that combines state-of-the-art literature with practice outside academia. Thus, researchers can summarize the specific topic and how it is operated by practitioners in various industries. In software engineering, applying MLR is a relatively new concept, and it is becoming popular as the findings from the literature are insightful [4, 26, 38]. Therefore, it is essential to include gray literature to discover and expand the research [31]. It is essential to incorporate gray literature and SLR together as they increase the scope of the research by broadening the perceptions, and thus, this adds value to the review process [31].

There are various guidelines for conducting SLR in software engineering. If we think about MLR, then the phases of the data collection process differ from the traditional systematic literature review process. More specifically, the process involved in searching for the MLR and assessing the quality of the sources differs from the SLR process [4].

In Figure 1, we have shown how the research process has been conducted. We have followed Garousi et al. [26] to plan, conduct, and report the findings after analyzing the data. While planning the MLR, first, we need to understand why we need to conduct the literature review, then define goals and address the research question. In the second stage, we conduct the process based on the search process, source selection, design of data extraction, data extraction, and data synthesis. In the final stage of reporting the multivocal literature review, we summarize the extracted data and report the findings. By following the process, there is substantial knowledge in the research area for DevOps practices and the impact of Critical success factors on DevOps performance [26]. Such literature might generate challenges as the evidence is based on experience, perceptions, and opinions. For this research, we will follow a search process similar to a Systematic literature review with some inclusion and exclusion criteria on the topic [31]. We use Google Search, Google Scholar, Scopus, IEEE Xplore and ACM Digital Library to obtain the search results.

Figure 1
Figure 1: Multivocal Literature Review steps followed for the research process (adapted from [4, 26])

3.2 Research questions

By considering the viewpoints of the professionals, practitioners, and academics our goal is to align the thoughts and create attentions that summarize the uncovered areas and scopes of research. To address this issue we have two research questions.

  • What are the critical success factors of DevOps discussed in multivocal literature?
  • Does the thought of MVLR align with the academic perspective and understanding of CSF?

3.3 Motivation of the research

Software Engineering companies are adopting and implementing DevOps practices, which has become a common phenomenon nowadays. For a successful implementation of DevOps management professionals of the company should have supporting and adequate information about the technological adoption and changes so that they can handle issues that increase the efficiency and the company's success can be assessed too [59].

However, some concepts, relationships, practices, and cultures are still not well established and in some cases not defined clearly by the practitioner's community. As this topic is mainly focusing on industries and various companies reports and concept summaries will be used for this research analysis and further exploration (state of the DevOps art). Thus, a multivocal literature review can identify the success factors and expand the source diversity for DevOps practices in various industries.

3.4 Search protocol

We have adopted the systematic literature review procedure for the study protocol and used the mass material available on the net. Before the search started, the keywords were selected and identified. For the search procedure, the authors have performed a structured search in Google Scholar and Google search engines including multiple databases and data libraries. For the literature review, we have collected data in two stages. We have used the same search string for both white and grey literature to get maximum information regarding DevOps critical success factors from both kinds of literature. For Collecting data for the former we have used multiple databases, and for the latter, we have followed the same search query strings. The search query we have implemented is:

(devops OR "development and operation") AND ("success factors" OR failure OR barriers OR drivers)

3.4.1 Search process for white literature (WL). For the search process of white literature, we have looked into various databases and libraries which has a good number of journals and conference papers. Those include Scopus, IEEE Xplore, ACM Digital Library and Google Scholar. These databases were selected based on the study's relevance and scope. We conducted the search process at the end of 2022. After inclusion and exclusion criteria we could get 208 items. Finally, we could consider 36 items after inclusion and exclusion criteria, abstract screening, and full-text assessment, which were most relevant for the research.

The inclusion criteria for the search are (1) Studies that describe DevOps success and failure, (2) Studies that have empirical evidence, and (3) Studies that are written in English. The exclusion criteria for the search are (1) Theoretical or literature review-based studies; (2) Studies written in a language other than English; (3) Full text that was unable to be downloaded.

Table 1: Number of search results elicited from database
Scientific Database Dataset search with the string No of paper finally selected
Scopus 422 10
IEEE Xplore 92 6
ACM library 1,692 9
Google Scholar 1,890 11
Google search 478 27

3.4.2 Search process for gray literature (GL). We have used the Google search engine based on defined search criteria and explored web resources available. We manually evaluated the items based on the inclusion and exclusion criteria for the selection process. The inclusion criteria for the search are (1) The items are available publicly and cover critical success factors, (2) The content of the document is meaningful, and (3) Real companies publish the item.

The exclusion criteria of the search are (1) The item does not consist only of technical or architectural aspects of DevOps implementation, (2) The item that does not explain Critical success factors elaborately, (3) The items that do not cover industrial evidence with professionals’ viewpoints. Finally, as the outcome, we got 70 items of gray literature, which contained 27 items shorted for data extraction and further analysis.

3.4.3 Data extraction and Analysis. After completing the search, we got 63 items that combined Gray and White literature. We have tried to extract related information to answer four research questions for DevOps critical success factors in organizations. For data extraction, at first, we have manually given input with related information available from publications. In the next stage, we analyzed the information collected from publications. At the final stage, we thoroughly reviewed all items so that no relevant information was left for reporting.

4 RESULTS

In this section, we discuss the overall results of the literature review. Then, we discuss two research questions and their findings.

4.1 MLR reporting

In this section we are going to discuss two research questions. The first question address about critical success factors regarding multivocal literature review. The second question is related to authors’ perspectives and understanding of critical success factors and its alignment with authors’ views.

4.1.1 RQ1: What are the critical success factors of DevOps based on multivocal literature?. According to Hernández et al [27] gray literature is essential for research because it gives a better understanding of automation, quality assurance, and testing of the requirements management process. Our findings have identified some critical success factors based on a multivocal literature review search. There could be several factors that could impact the performance of the organizations. Some of the essential factors could be culture, automation, continuous integration and deployment, monitoring, security, leadership, standardization, etc. Many sub-factors are involved in DevOps operations, but it is impossible to highlight them all here.

In retrospect, it is clear that software development was previously a laborious and time-consuming task. Programmers had to manually input lines of code and were often lacking in the necessary resources and technological support, which made their work quite challenging. As [66] points out, developers had to devote a considerable amount of time to handling builds on a daily basis, causing further operational overhead.

The agile movement in software development emphasizes the value of building strong relationships and fostering a sense of community among developers. Instead of relying solely on institutionalized processes and development tools, agile practices prioritize the formation of close-knit teams within a collaborative working environment [1]. These teams are committed to producing continuously tested and functional software through regular release cycles, sometimes even daily. Developers are encouraged to focus on creating code that is technically advanced, easy to comprehend, and requires minimal documentation, which helps streamline the development process and maintain smooth operations [1, 51].

According to Moe et al. four basic principles express agile methods [45]. Those include individuals and interaction with tools, Comprehensive documentation with working software, customer collaboration, and response to changes by following a plan [1, 45]. According to Dybå and Dingsøyr, There is a limitation while working on an agile team as the team members are not interchangeable, leading to various consequences for the projects and how it is managed [19].

On the other hand, DevOps culture is the increased collaboration between the Dev and Ops teams. Basically, DevOps culture emphasizes collaboration, better communication, and ongoing learning of the technology [8, 40, 57]. Any organization that wants to achieve DevOps success must adopt the organizational culture within teams. There could be several important cultural changes that lead to shared responsibilities, no silos between teams, and adjusting resource structures for the operations teams so that they can be involved from the beginning of the development process [4, 28].

We have got several factors for DevOps culture. Those include cross team collaboration [4], communication, Support learning culture [4], team and process efficiency [7, 18], Lack of understanding and reluctance in teams [60], Mutuality, trust and team commitment [5, 60], Efficient collaboration [8, 63], cross-functional teams integration [7, 64], source code control [6, 15], cultural norms values and mutual respect [32], cultural impacts [48], diversity and knowledge absorption [65] etc.The best part of having a DevOps culture is the elimination of silos between the development teams and operations teams, which improves the work culture in teams.

Automation is another integral part of DevOps operations as that facilitates the software development process to achieve faster and more efficient software delivery. Automation tools help with testing, configuration, and deployment and reduce human error in the process. For the software delivery process, the effective use of CI/CD pipelines is crucial. For DevOps, the implementation of CI/CD pipelines continuously integrates and deploys the software.

There are factors we came across which impact the whole process of automation. Those factors includes improvement in test automation [6, 53], Verification [8, 11], Specific tools to automation [39], Automatic testing tools and techniques [2], Development and test automation [6], Automation and process automation [65] etc.

Monitoring and feedback are other essential aspects of DevOps. The DevOps team should monitor the process continuously so that the performance of the software can meet user requirements. There are various tools available for software development. According to Lawkatare et al., in the embedded system, having access to monitoring data could increase the performance, and the company could get great benefits with this [8, 40]. DevOps teams should use a tool that could maintain consistency, and efficiency with a faster delivery time for the software development life cycle [5, 7, 8].

Team leadership and collaboration are essential to ensure the successful implementation and adoption of DevOps practices. A team with a good leader encourages and supports team members and this balances the cultural shift towards DevOps. Thus, the necessary resources and infrastructure are well implemented when there is better leadership in teams. There are some team leadership factors that impact on the company's performance. Those includes collaboration between dev and ops [8], departments alignment problem [17], Communication and inflexibility issues [18], communication structures [60], miscommunication between development and operations [18], help from cross-organizational teams [18], lack of lead person in teams [3], silos emphasizing [18], setting a clear role for the team [65], company pressure [60].

The main objective of software security is to understand beforehand the motives of attackers and foresee and imagine how to handle such situations if required. When software is developed the thought that works is just to build it and hope that the application will work normally. when security issues are combined with the software building process then it becomes harder for the attackers to create threats for the system [10, 44].

DevSecOps is an essential part of DevOps. Security should be integrated into the software development process from the beginning. This will ensure that the software will be delivered with fewer vulnerabilities. For better output, the security testing should be integrated into CI/CD pipelines and should be automated. According to Mao et al [43] DevSecOps provides security continuously that improves continual learning and continual efficiency of product security. Mao et al. suggested that sharing responsibilities in teams is crucial for the team members because when every member is part of the value chain, then they are responsible for the security process from the beginning till the end [43]. From the gray literature, we have got some thoughts from the professionals that security should be involved earlier in the software development process. This mindset will accelerate the development and operations teams to have better security and understanding of the other functions in the process. According to McGraw, security should be placed at the beginning of the software process, and it is everyone's responsibility to maintain software security [10, 43, 44].

4.1.2 RQ2: Does the thought of MVLR align with the academic perspective and understanding of CSF?. The second question explains the author's perspective and academic understanding of Critical success factors and their alignment with the multi-vocal literature review. This can give a better understanding of DevOps practices and success factors that impact software development performance. Gray literature discusses more industrial and practitioner-oriented information, and academic literature is all about academic research based on various perspectives and viewpoints [4]. Our research thus add value for practitioners and academics in knowing the similarities among various success factors of DevOps practices. Based on the prior research by multivocal literature review, some publications discussed DevOps practices, adoption, and DevOps implementation practices in companies [5, 6, 7, 8, 12].

The gray literature shows that “critical success” or “DevOps critical success” was used frequently in gray documents. In the white literature, we have found that the academics emphasized critical success factors and challenges associated with DevOps practices. Prior studies also explained the importance of knowing the Critical success, which makes the software development process faster and more efficient. Academics have found various perspectives on DevOps practices’ success factors, and the researchers got some interesting findings for practitioners. Researchers reported their findings based on company practices, which are more generic concepts of DevOps practices. The study reveals an exciting insight into the difference in usage of terminology between the gray literature and scientific articles. The analysis of reports materials and Conferences, which are closer to the grey literature, shows a clear inclination towards the term DevOps Practice  [4]. In contrast, scientific articles tend to use the Capabilities and requirements of DevOps practices more frequently. This finding sheds light on the contrasting focus of practitioners and the scientific community. Practitioners are more focused on DevOps Practices, while the scientific community is more inclined to organize Capabilities in a way that abstracts more generic concepts applicable to building skills and enablers  [4, 8, 9].

Here, the multivocal literature review concept is well connected with the author's perspective regarding organizational critical success factors. Each success factor was analyzed based on the literature available for better understanding. According to the author, the word “Critical success factors” gives a high-level overview based on the specific context of DevOps. Sometimes, it is not easy to define the factors that are directly impacting the success of the software development process. According to our findings, several factors might impact the overall process of the software life cycle. The grey literature discusses KPIs, Cross-functional teams, automation, security, Project life cycle and service, Culture, legacy systems, application complexity, unclear plan, skillset, Budget, tools, DevOps environment, Leadership, standardization, Monitoring, feedback, and integration.

5 DISCUSSION AND FUTURE WORK

5.1 Discussion

It has been observed that industries have more extensively analyzed and explored the topic of DevOps than scientific research. To maintain good practices within the organization, team managers must have a clear understanding of the practices and elements that contribute to a project's success, and the necessary skills that team members must possess.

The team's managers need to have a clear idea about the practices, what are the elements that lead to a project's success, and the capabilities required by team members to maintain good practices in the organization [5, 7, 35]. The expected outcomes can contribute to better knowledge and understanding of DevOps issues and situations in the organization. By doing so, a better comprehension of DevOps issues and situations within the organization can be achieved [5, 7].

If we compare the traditional development model, DevOps is an addition that combines both teams, enables continuous deployment, and increases the delivery frequency. Still, there are some security risks involved in the process [6, 40, 43, 57]. According to Banerjee et al [10] there are 21 rules to maintain security for any software development process. At the same time, to understand DevOps practices in the companies, it is essential to understand the factors that increase the performance of the process and lead a project to success or failure [5, 6, 7, 8].

According to Kraemer et al.  [33], programmers sometimes overlook security due to a lack of time and a high workload. When a system has DevOps adoption, the software is deployed and delivered faster, and there is less possibility of unexpected mistakes. At the same time, if the production environment doesn't have sufficient security reviews, there is a high risk of vulnerabilities in the process [22]. The organization is now incorporating DevSecOps, which could be a great approach to maintain the security issues, and that helps to increase the quality of software during the lifecycle of software development [22, 46]. To maintain security issues and improve software quality during the software development lifecycle, organizations are increasingly turning towards DevSecOps as a viable approach [22, 46]

Our findings on DevOps success factors also suggest that there are several key elements that contribute to the successful implementation of DevOps practices in organizations. These include strong leadership, clear communication, cross-functional collaboration, automation, continuous improvement, and a culture of experimentation and innovation.

Strong leadership is important because it sets the tone for the organization and helps to create a shared vision for DevOps implementation. Clear communication is essential to ensure that everyone in the organization understands the goals and objectives of DevOps and is working towards the same outcomes. Cross-functional collaboration is critical because it breaks down silos and encourages different teams to work together towards a common goal. Automation helps to streamline processes and reduce the risk of errors, while continuous improvement ensures that DevOps practices are constantly being evaluated and refined. Finally, a culture of experimentation and innovation encourages teams to take risks, test new ideas, and learn from failures.

By focusing on these key success factors, organizations can increase their chances of successfully implementing DevOps practices and realizing the benefits of faster, more reliable software delivery.

5.2 Threats to validity

In the multivocal literature review, we observed some validity issues. We have followed Petersen et al. for the threads to validity. We have observed theoretical validity concerns for the search [50]. There could be a selection bias for the search, and to mitigate this issue, we have applied different sources and used inclusion and exclusion criteria. According to our understanding, video content could have limitations as we could not include videos to collect information for the research. As this is multivocal literature, the materials used for data analysis were not published and peer-reviewed, meaning the materials were not similar to academic papers. This is a significant risk. According to Garousi et al.  [26], to mitigate this, we used the recommended procedures and followed the steps recommended by the author. In our research, we have tried to maintain a systematic process so that the primary data and the data we have collected have a common ground for data extraction.

For the multivocal literature review, there could be publication bias because the collected data is not exclusively peer-reviewed and published materials. In the blog posts, there could be a biased opinion regarding DevOps practices, adoption, and new recommendation for professionals. To mitigate this issue, we carefully went through all documents and tried to stay focused on our specific topic. Two researchers tried to compare different information by making the comparison.

5.3 Future work

First, For future research, there could be opportunities to know critical success factors along with more sub-levels and classification of factors associated with the process. As the companies are adopting and implementing DevOps there could be many unknown success factors that would be identified with new searches. DevOps is evolving and many changes are taking place.

Second, there could be various challenges and issues while moving from traditional development techniques to new concepts of DevOps. There could be more research on Critical success factors that might help to mitigate DevOps’ different issues for organizations’ future progress.

Third, there is no MLR research yet done on DevOps critical success factors. Thus the authors would like to perform research for identifying success factors based on white literature.

Fourth, the authors could conduct a survey on companies to expand the study of Critical success factors. It would be beneficial to know company practices, DevOps challenges, and collaboration between Dev and Ops teams.

Fifth, DevOps and its practices are extending and evolving rapidly. It is important that in Software Engineering, DevOps concepts should be aligned properly. Various research can be conducted so that the adoption process of DevOps practices becomes easy to understand and execute.

Sixth, In future studies, we can try to create an artifact with the design science approach as we have various information related to DevOps concepts and practices. If we could map the results from concepts and practices that could provide a new ontology that synthesizes the concepts found from the study.

Seventh, we believe that there is a scope to compare agile methods and DevOps practices, how the critical success factors impact the performance of software development, and what the similarities and dissimilarities between agile and DevOps methods are based on academic and industrial literature.

Eighth, In the extended version of the paper, we are planning to do more research and answer more research questions related to DevOps critical success factors benefits, risks, and challenges in organizations. This could help us to broaden our views and insights for the research.

Finally, there are huge materials on the internet regarding critical success factors, and the scope to use most of them is limited and challenging. In the extended version, we are aiming to cover the rest of the materials so that the findings will give better findings and insights for a multivocal literature review.

6 CONCLUSION

Our literature review has identified several crucial factors for the success of DevOps, including culture, automation, integration, deployment, monitoring, feedback, standardization, leadership, and security. Although DevOps is gaining attention, it is essential to understand practitioners’ perspectives. Our research can broaden DevOps practices and perspectives, enhance academic knowledge, and strengthen success factors. DevOps operations face challenges for critical success factors in organizations. Our paper discusses these factors, implementation challenges, and professional viewpoints. Conducting interviews and surveys with DevOps professionals could provide opportunities for learning and conducting empirical studies [37].

REFERENCES

  • Pekka Abrahamsson, Outi Salo, Jussi Ronkainen, and Juhani Warsta. 2017. Agile software development methods: Review and analysis. arXiv preprint arXiv:1709.08439 (2017).
  • Muhammad Azeem Akbar, Sajjad Mahmood, Muhammad Shafiq, Ahmed Alsanad, Abeer Abdul-Aziz Alsanad, and Abdu Gumaei. 2020. Identification and prioritization of DevOps success factors using fuzzy-AHP approach. Soft Computing (2020), 1–25.
  • Betool Alnamlah, Sara Alshathry, Norah Alkassim, and Nor Jamail. 2021. The necessity of a lead person to monitor development stages of the DevOps pipeline. Indonesian Journal of Electrical Engineering and Computer Science 21 (01 2021), 348. https://doi.org/10.11591/ijeecs.v21.i1.pp348-353
  • Ricardo Manuel Duarte Amaro, Ruben Pereira, and Miguel Mira da Silva. 2022. Capabilities and practices in DevOps: a multivocal literature review. IEEE Transactions on Software Engineering (2022).
  • Nasreen Azad. 2022. Understanding DevOps critical success factors and organizational practices. In 2022 IEEE/ACM International Workshop on Software-Intensive Business (IWSiB). IEEE, 83–90.
  • Nasreen Azad and Sami Hyrynsalmi. 2021. What Are Critical Success Factors of DevOps Projects? A Systematic Literature Review. In Software Business(Lecture Notes in Business Information Processing, Vol. 434), Xiaofeng Wang, Antonio Martini, Anh Nguyen-Duc, and Viktoria Stray (Eds.). Springer, Cham. https://doi.org/10.1007/978-3-030-91983-2_17
  • Nasreen Azad and Sami Hyrynsalmi. 2022. DevOps Challenges in Organizations: Through Professional Lens. In Software Business: 13th International Conference, ICSOB 2022, Bolzano, Italy, November 8–11, 2022, Proceedings. Springer, 260–277.
  • Nasreen Azad and Sami Hyrynsalmi. 2023. DevOps critical succes factors—A systematic literature review. Information and Software Technology (2023), 107150.
  • Nasreen Azad and Sami Hyrynsalmi. 2023. DevOps critical succes factors—A systematic literature review. Information and Software Technology (2023), 107150.
  • C Banerjee and SK Pandey. 2009. Software security rules, SDLC perspective. arXiv preprint arXiv:0911.0494 (2009).
  • Walid Ben Mesmia, Mohamed Escheikh, and Kamel Barkaoui. 2021. DevOps workflow verification and duration prediction using non-Markovian stochastic Petri nets. Journal of Software: Evolution and Process 33, 3 (2021), e2329.
  • Cor-Paul Bezemer, Simon Eismann, Vincenzo Ferme, Johannes Grohmann, Robert Heinrich, Pooyan Jamshidi, Weiyi Shang, André van Hoorn, Monica Villavicencio, Jürgen Walter, et al. 2019. How is performance addressed in DevOps?. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. 45–50.
  • Moroz Bogdan, Andrey Saltan, and Sami Hyrynsalmi. 2023. Product Ops: Understanding and Defining an Emerging Discipline. In Int.Conf. on Software Product Management. GI, 11.
  • Lianping Chen. 2017. Continuous delivery: overcoming adoption challenges. Journal of Systems and Software 128 (2017), 72–86.
  • Gerry Gerard Claps, Richard Berntsson Svensson, and Aybüke Aurum. 2015. On the journey to continuous deployment: Technical and social challenges along the way. Information and Software technology 57 (2015), 21–31.
  • Patrick Debois. 2008. Agile infrastructure and operations: how infra-gile are you?. In Agile 2008 Conference. IEEE, 202–207.
  • Jessica Díaz, Daniel López-Fernández, Jorge Pérez, and Ángel González-Prieto. 2021. Why are many businesses instilling a DevOps culture into their organization?Empirical Software Engineering 26, 2 (2021), 1–50.
  • Jessica Díaz, Jorge E Perez, Agustin Yague, Andrea Villegas, and Antonio de Antona. 2019. Devops in practice–a preliminary analysis of two multinational companies. In International Conference on Product-Focused Software Process Improvement. Springer, 323–330.
  • Tore Dyba and Torgeir Dingsoyr. 2009. What do we know about agile software development?IEEE software 26, 5 (2009), 6–9.
  • Andrej Dyck, Ralf Penners, and Horst Lichter. 2015. Towards definitions for release engineering and DevOps. In 2015 IEEE/ACM 3rd International Workshop on Release Engineering. IEEE, 3–3.
  • Floris MA Erich, Chintan Amrit, and Maya Daneva. 2017. A qualitative study of DevOps usage in practice. Journal of software: Evolution and Process 29, 6 (2017), e1885.
  • Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. 2013. Development and deployment at facebook. IEEE Internet Computing 17, 4 (2013), 8–17.
  • Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda. Journal of Systems and Software 123 (2017), 176–189.
  • Nicole Forsgren and Jez Humble. 2016. The role of continuous delivery in IT and organizational performance. Forsgren, N., J. Humble (2016)." The Role of Continuous Delivery in IT and Organizational Performance." In the Proceedings of the Western Decision Sciences Institute (WDSI) (2016).
  • Martin Fowler and Matthew Foemmel. 2006. Continuous integration.
  • Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology 106 (2019), 101–121.
  • Rogelio Hernández, Begoña Moros, and Joaquín Nicolás. 2023. Requirements management in DevOps environments: a multivocal mapping study. Requirements Engineering (2023), 1–30.
  • Jez Humble and David Farley. 2010. Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education.
  • Jez Humble and Gene Kim. 2018. Accelerate: The science of lean software and devops: Building and scaling high performing technology organizations. IT Revolution.
  • Jon Iden, Bjørnar Tessem, and Tero Päivärinta. 2011. Problems in the interplay of development and IT operations in system development projects: A Delphi study of Norwegian IT experts. Information and Software Technology 53, 4 (2011), 394–406.
  • Barbara A Kitchenham. 2012. Systematic review in software engineering: where we are and where we should be going. In Proceedings of the 2nd international workshop on Evidential assessment of software technologies. 1–2.
  • Gwendolyn L Kolfschoten, Gert-Jan de Vreede, Robert O Briggs, and Henk G Sol. 2010. Collaboration ‘engineerability’. Group Decision and Negotiation 19, 3 (2010), 301–321.
  • Sara Kraemer, Pascale Carayon, and John Clem. 2009. Human and organizational factors in computer and information security: Pathways to vulnerabilities. Computers & security 28, 7 (2009), 509–520.
  • JANE LAUDON LAUDON. 2017. PEARSON MYLAB MIS WITH PEARSON ETEXT-INSTANTACCESS-MANAGEMENT INFORMATION SYSTEMS:... Managing the Digital Firm, Global Edition. PEARSON EDUCATION LIMITED.
  • Eero Laukkanen, Juha Itkonen, and Casper Lassenius. 2017. Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology 82 (2017), 55–79.
  • Timo OA Lehtinen, Mika V Mäntylä, Jari Vanhanen, Juha Itkonen, and Casper Lassenius. 2014. Perceived causes of software project failures–An analysis of their relationships. Information and Software Technology 56, 6 (2014), 623–643.
  • Joel K Leidecker and Albert V Bruno. 1984. Identifying and using critical success factors. Long range planning 17, 1 (1984), 23–32.
  • Iraj Lohrasbinasab, Prameet Bhakta Acharya, and Ricardo Colomo-Palacios. 2020. BizDevOps: a multivocal literature review. In Computational Science and Its Applications–ICCSA 2020: 20th International Conference, Cagliari, Italy, July 1–4, 2020, Proceedings, Part VI 20. Springer, 698–713.
  • Welder Pinheiro Luz, Gustavo Pinto, and Rodrigo Bonifácio. 2018. Building a collaborative culture: a grounded theory of well succeeded devops adoption in practice. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10.
  • Lucy Ellen Lwakatare, Teemu Karvonen, Tanja Sauvola, Pasi Kuvaja, Helena Holmström Olsson, Jan Bosch, and Markku Oivo. 2016. Towards DevOps in the embedded systems domain: Why is it so hard?. In 2016 49th hawaii international conference on system sciences (hicss). IEEE, 5437–5446.
  • Ruth W Macarthy and Julian M Bass. 2020. An empirical taxonomy of DevOps in practice. In 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 221–228.
  • Simo Mäkinen, Marko Leppänen, Terhi Kilamo, Anna-Liisa Mattila, Eero Laukkanen, Max Pagels, and Tomi Männistö. 2016. Improving the delivery cycle: A multiple-case study of the toolchains in Finnish software intensive enterprises. Information and Software Technology 80 (2016), 175–194.
  • Runfeng Mao, He Zhang, Qiming Dai, Huang Huang, Guoping Rong, Haifeng Shen, Lianping Chen, and Kaixiang Lu. 2020. Preliminary findings about devsecops from grey literature. In 2020 IEEE 20th international conference on software quality, reliability and security (QRS). IEEE, 450–457.
  • Gary McGraw. 2004. Software security. IEEE Security & Privacy 2, 2 (2004), 80–83.
  • Nils Brede Moe, Torgeir Dingsøyr, and Tore Dybå. 2008. Understanding self-organizing teams in agile software development. In 19th australian conference on software engineering (aswec 2008). IEEE, 76–85.
  • Vaishnavi Mohan and Lotfi Ben Othmane. 2016. Secdevops: Is it a marketing buzzword?-mapping research on security in devops. In 2016 11th international conference on availability, reliability and security (ARES). IEEE, 542–547.
  • Haavard Myrbakken and Ricardo Colomo-Palacios. 2017. DevSecOps: a multivocal literature review. In Software Process Improvement and Capability Determination: 17th International Conference, SPICE 2017, Palma de Mallorca, Spain, October 4–5, 2017, Proceedings. Springer, 17–29.
  • Kristian Nybom, Jens Smeds, and Ivan Porres. 2016. On the impact of mixing responsibilities between devs and ops. In International Conference on Agile Software Development. Springer, 131–143.
  • Marta Olszewska and Marina Waldén. 2015. DevOps meets formal modelling in high-criticality complex systems. In Proceedings of the 1st international workshop on quality-aware DevOps. 7–12.
  • Kai Petersen and Claes Wohlin. 2009. A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. Journal of systems and software 82, 9 (2009), 1479–1490.
  • Minna Pikkarainen, Jukka Haikara, Outi Salo, Pekka Abrahamsson, and Jari Still. 2008. The impact of agile practices on communication in software development. Empirical Software Engineering 13 (2008), 303–337.
  • Luís Prates, João Faustino, Miguel Silva, and Rúben Pereira. 2019. Devsecops metrics. In Information Systems: Research, Development, Applications, Education: 12th SIGSAND/PLAIS EuroSymposium 2019, Gdansk, Poland, September 19, 2019, Proceedings 12. Springer, 77–90.
  • Leah Riungu-Kalliosaari, Simo Mäkinen, Lucy Ellen Lwakatare, Juha Tiihonen, and Tomi Männistö. 2016. DevOps adoption benefits and challenges in practice: a case study. In International conference on product-focused software process improvement. Springer, 590–597.
  • John F Rockart. 1979. Chief executives define their own data needs.Harvard business review 57, 2 (1979), 81–93.
  • Pilar Rodríguez, Alireza Haghighatkhah, Lucy Ellen Lwakatare, Susanna Teppola, Tanja Suomalainen, Juho Eskeli, Teemu Karvonen, Pasi Kuvaja, June M Verner, and Markku Oivo. 2017. Continuous deployment of software intensive products and services: A systematic mapping study. Journal of systems and software 123 (2017), 263–291.
  • Mary Sánchez-Gordón and Ricardo Colomo-Palacios. 2018. A multivocal literature review on the use of DevOps for e-learning systems. In Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality. 883–888.
  • Mojtaba Shahin. 2015. Architecting for devops and continuous deployment. In Proceedings of the ASWEC 2015 24th Australasian Software Engineering Conference. 147–148.
  • Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. 2017. Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE access 5 (2017), 3909–3943.
  • Jens Smeds, Kristian Nybom, and Ivan Porres. 2015. DevOps: a definition and perceived adoption impediments. In Agile Processes in Software Engineering and Extreme Programming: 16th International Conference, XP 2015, Helsinki, Finland, May 25-29, 2015, Proceedings 16. Springer, 166–177.
  • Darja Šmite, Nils Brede Moe, and Javier Gonzalez-Huerta. 2021. Overcoming cultural barriers to being agile in distributed teams. Information and Software Technology 138 (2021), 106612.
  • Daniel Ståhl and Jan Bosch. 2014. Automated software integration flows in industry: A multiple-case study. In Companion Proceedings of the 36th International Conference on Software Engineering. 54–63.
  • Michiel Van Belzen, Jos Trienekens, and Rob Kusters. 2019. Critical success factors of continuous practices in a DevOps context. In 28th International Conference of Information Systems Development (ISD 2019).
  • Johannes Wettinger, Uwe Breitenbücher, and Frank Leymann. 2014. Devopslang–bridging the gap between development and operations. In European Conference on Service-Oriented and Cloud Computing. Springer, 108–122.
  • Anna Wiedemann, Manuel Wiesche, Heiko Gewald, and Helmut Krcmar. 2020. Understanding how DevOps aligns development and operations: a tripartite model of intra-IT alignment. European Journal of Information Systems 29, 5 (2020), 458–473.
  • Changbyung Yoon, Keeeun Lee, Byungun Yoon, and Omar Toulan. 2017. Typology and success factors of collaboration for sustainable growth in the IT service industry. Sustainability 9, 11 (2017).
  • Liming Zhu, Len Bass, and George Champlin-Scharff. 2016. DevOps and its practices. IEEE software 33, 3 (2016), 32–34.

CC-BY license image
This work is licensed under a Creative Commons Attribution International 4.0 License.

EASE 2024, June 18–21, 2024, Salerno, Italy

© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-1701-7/24/06.
DOI: https://doi.org/10.1145/3661167.3661236