Research for Practice: The DevOps Phenomenon

 

A traditional software company releases its flagship product maybe every few years. Each release can include hundreds of new features and improvements. Because releases are infrequent, users can grow impatient waiting for each new release and are thankful when it finally arrives.

Disappointment sets in, however, when bugs are found and features don’t work as expected. Under great stress and with great turmoil, an emergency release is produced and put into production (hurried through the regular release process, often achieved by skipping tests), which has still more bugs, and the process repeats with more emergency releases, leading to more frustration, stress, and disappointment. Worse yet, new business opportunities are missed or ignored because of doubt, uncertainty, and distrust in the IT department’s ability to deliver value.

Isn’t there a better way?

Such practices are a thing of the past for companies that subscribe to the DevOps method of software development and delivery. New releases are frequent: often weekly or daily. Bugs are fixed rapidly. New business opportunities are sought with gusto and confidence. New features are released, revised, and improved with rapid iterations. In one case study, a company was able to provide a new software feature every 11 seconds.¹⁷

Which of these software teams would you rather be? Which of these companies will win during their industry’s digital transformation?

DevOps presents a strategic advantage for organizations when compared with traditional software-development methods (often called phase-gate or waterfall.⁷ Leadership plays an important role during that transformation. In fact, Gartner predicts that CIOs who haven’t transformed their teams’ capabilities by 2020 will be displaced.⁹

Back to Top

Promises and Challenges for Digital Transformation

For organizations hoping to capture market share and deliver value faster (or even just deliver software more safely and securely), DevOps promises both speed and stability.⁴ It has developed as a prominent phenomenon of digital transformation in modern organizations that use software to deliver value to their customers in industries including banking, retail, and even manufacturing. DevOps combines activities of software development and delivery to enhance the speed of getting new software features to customers. This leads to higher customer satisfaction and profitability, which are important outcomes at the organization level. It also leads to important team-level outcomes such as better collaboration among different departments (such as developers, testers, and IT operations) and improved work-life balance.

Executing a successful DevOps transformation isn’t without its challenges. Organizations and software products vary in maturity and implementation, making transformation efforts difficult to design and deploy across teams and organizations. Most importantly, for DevOps to truly deliver value, it must include more than just tooling and automation—so simply purchasing and installing a solution is not sufficient. As outlined here, DevOps includes culture, process, and technology. Indeed, success stories abound: Companies such as Kaiser Permanente, Capital One, Target, Starbucks, and ING have adopted DevOps methods, allowing them to deliver software for key applications in just seconds.

DevOps enhances automation from applications to infrastructure provisioning. Continuous delivery supports automation and enables faster time to market and agile software development with fast feedback cycles. As the phenomenon is relatively new in practice, practitioners report on struggles with issues such as leadership and cultural transformation, implementing continuous delivery pipelines, and integrating a culture of collaboration in team settings.⁸ Each of these areas would benefit from examination and guidance by formal research to test and augment the valuable experiences of those in industry.

Back to Top

A Move Away from Traditional Project Management

• DevOps methodology is marked by a change in how software delivery is treated: moving from a discrete project to an ongoing product. The traditional way of delivering software (referred to previously as phase-gate or waterfall) happens with the help of project management. In this paradigm, the project typically “ends” with the first major release of the new software or a set of new features delivered in a major incremental release. In general, the project team dissolves following a new release, and the responsibility for running the system is then “thrown over the wall” to operations. The operations team takes over responsibility for further changes and incident management. This leads to several problems:
• Developers are not responsible for running the system they built and therefore do not understand if tradeoffs appear in creating and running the system, notably in the scalability and reliability of the software. This can lead to the same problems being perpetuated in future software releases, even if they are well understood by the operations team.
• The operations team is responsible for maintaining a highly reliable and stable system. Each new line of software deployment introduces change, and therefore leads to instability. This leads to a mismatch in incentives, where accepting new software from developers introduces risk (instability) and uncertainty (because less or no visibility into the development process gives them little insight into the software they are inheriting). Even if new components are of high quality, all new code adds complexity to the system and risk that the software was not developed with scalability and reliability in mind—key factors that operations must address and support in new software, as well as existing software.
• Stakeholders upstream (that is, earlier in the development process) from the production environment, including developers and the business, are not able to receive any feedback about performance until the first complete release. This generally takes place several months after project approval. Furthermore, not all features in software deliver value, and speeding up feedback to the development team and business allows for faster iteration to refine the software. This leads to wasted time and resources on features that don’t ultimately deliver value. For example, research from Microsoft shows that only one-third of well-designed features delivered value to the customer.¹⁴

In contrast, DevOps calls for a shift to product-based management. Practically, this means there is no more “end date” to projects, and teams instead deliver features—and therefore value—continuously. An important part of achieving this is integrating teams throughout the value stream, from development through operations; some organizations are even including business stakeholders. In this model, software is a product that is maintained as a product, with delivery and value metrics being tracked by the business continuously.

Back to Top

State-of-the-Art Research and Practice on DevOps

One of the biggest challenges (and complaints!) in industry is the lack of a formal definition for DevOps. Many practitioners argue that this is intentional, because it allows teams and organizations to adopt a definition that works for them. In addition, they point out that having a formal definition of agile (coded in the Agile Manifesto) hasn’t solved the problem of definition sprawl, and the resulting confusion around what is truly meant when an organization says it is “going agile” still plagues the industry. This lack of understanding can be challenging, so we present some common definitions for reference here.

• “DevOps is a software development and delivery methodology that provides … increased speed and stability while delivering value to organizations.”⁴
• “DevOps, whether in a situation that has operations engineers picking up development tasks or one where developers work in an operations realm, is an effort to move the two disciplines closer.”¹⁶
• “DevOps, a compound of ‘development’ and ‘operations,’ is a software development and delivery approach designed for high velocity.”¹⁷

And yet, these definitions focus on the outcome (value through speed and stability, via Forsgren⁴) or the foundations of the discipline (bringing together development and operations, via Roche¹⁶ or Sebastian et al.¹⁷ If one wants to understand the components of DevOps methods, perhaps the most common presentation of these is summarized as CALMS: culture, automation, lean, measurement, and sharing.^6,12 Here is a brief definition of each component:

• Culture. Integration of mutual trust, willingness to learn, continuous improvement, constant flow of information, open-mindedness to changes, and experimentation between developers and operations.
• Automation. Implementing deployment pipelines with a high level of automation (most notably continuous integration/continuous delivery) and comprehensive test automation.
• Lean. Applying lean principles such as minimization of work in progress, as well as shortening and amplification of feedback loops to identify and minimize value flow breaks to increase efficiency.
• Measurement. Monitoring the key system metrics such as business or transactions metrics and other key performance indicators.
• Sharing knowledge in the organization and across organizational boundaries. Team members should learn from each other’s experiences and pro-actively communicate.

Back to Top

An Interpretation of CALMS

This article uses the five core elements of CALMS as a framework.

Culture. Working as part of a Dev Ops team requires a culture of collaboration within a cross-functional team setting. In its ideal state, DevOps uses a so-called cross-functional team, which means that groups made up of developers, testers, quality assurance professionals, and IT operations engineers all work together to develop and deliver software. In this way, they are familiar with each others’ work and challenges (a common phrase to describe this is “to have empathy”), which helps them create and maintain better software. For example, because they see the challenges in maintaining scalable and reliable infrastructure encountered by operations professionals, developers write more scalable and reliable code in collaboration with operations staff.

Automation. This principle requires a suite of DevOps tools.^2,13 The following are a few examples of available automation tooling:

• Build. Tools for the software development and service life cycle, including compiling code, managing dependencies, creating documentation, conducting tests, and deployments of an application in different stages.
• Continuous integration. Constant testing of new software components.
• Release automation. Packaging and deploying a software component from development across all environments to production.
• Version control. Managing changes to the program and collecting other information. Version control is a component of configuration management.
• Test automation. Using software to execute tests and repeating activities.
• Configuration management. Reducing production provisioning and configuration maintenance with the help of reproducing the production system on development stages.
• Continuous delivery. A combination of principles, practices, and patterns designed to make uninterrupted deployments.
• Logging. Traces are essential for application management; everything must be logged.
• Monitoring. Identification of infrastructure problems before the customers notice them. Monitoring storage, network traffic, memory consumption, etc.

Continuous integration and delivery reduce the cost and risk of releasing software. They do this by combining automation and good practices to consistently, reliably, and repeatably perform work (such as tests and builds) that enables fast feedback and builds quality in during the software-delivery process. This helps teams deliver features faster and more reliably and, in turn, achieve faster value delivery for the organization. The highest-performing teams are able to deploy 46 times more frequently with 2,555 times shorter lead times than low performers. Failure rates are seven times less, and they are able to recover 2,604 times faster than their lower-performing peers.⁸

Lean. “Building quality in”—referenced earlier—is a key tenet in DevOps, and a principle also found in lean. Applied to DevOps, this means teams look for opportunities to remove waste, leverage feedback loops, and optimize automation.

---

Companies such as Kaiser Permanente, Capital One, Target, Starbucks, and ING have adopted DevOps methods, allowing them to deliver software for key applications in just seconds.

---

Let’s look at an example. DevOps teams differ in size and product responsibility. In some models, a single team conducts all software development and delivery activities, including development, testing, delivery, and maintenance. They are ultimately responsible for the complete software-delivery life cycle of the software products (and may be responsible for more than one product), delivering value to the business. Lean processes allow for quick iterations and feedback throughout the development and delivery process to improve quality and build faster and more reliable systems. Examples include working in small batches to enable fast flow through the development pipeline, limiting work in process, fixing errors as they are discovered vs. at the end, and “shifting left” on security input.

These practices may sound familiar; similar practices have driven quality and value in manufacturing. (For a great story about lean manufacturing, check out episode 561 of This American Life,¹¹ which discusses NUMMI (New United Motor Manufacturing Incorporated), the joint venture between Toyota and GM in Fremont, CA.)

Measurement is another core aspect of DevOps. The ability to monitor and observe systems is important, because software development and delivery are essentially dealing with an invisible inventory that interacts in complex ways that cannot be observed. (This is in contrast to traditional physical manufacturing systems such as an automobile assembly line, described in the NUMMI case.)

Through effective monitoring, teams are able to track, watch, measure, and debug their systems throughout the software-delivery life cycle. It should be noted that metrics are also a tool for quality assurance, and measurements from several sources should be leveraged.⁹

Sharing of knowledge and information enables successful DevOps teams and helps amplify their success. By sharing practices—both successes and failures—within teams, across the organization, and across the industry, teams benefit from the learning of others and improve faster. While others have pointed out that sharing is possible in any domain and any methodology, DevOps has adopted this as a cultural norm, and many in the industry report that the field is much more collaborative than their prior work in tech.

Internal collaboration may include work shadowing or job swapping: developers are involved in operations and maintenance activities (for example, developers may even “take the pager”), and operations engineers rotate in to development and test roles, learning essential components of design and test work. In many cases, all cross-functional team members participate in the same meetings, which gives them shared context. Cross-industry sharing often takes places at conferences, with dozens of DevOps Days and other community-organized events sprouting up around the world.

---

As many practitioners note, managing and enabling the cultural changes inside an organization can be a more difficult and important challenge than implementing technical changes.

---

The application of these principles leads to better outcomes: for individuals (seen in reduced burnout and greater job satisfaction), for teams (seen in better software delivery outcomes and better team cultures), and for organizations (seen in improved performance in measures such as profitability, productivity, customer satisfaction, and efficiency.^7,21

Although DevOps has been an important movement in industry for more than a decade, it has not received much attention from the academic community until recently. And while CALMS principles are not always referred to using these terms, they do appear in existing research (for example, Fitzgerald and Stol³).

Back to Top

Research Summaries

The core insights of some prior Dev Ops-related research follow:

Journal Articles

• Fitzgerald and Stol. The Journal of Systems and Software³

Presents a continuous software-engineering pipeline and a research agenda for different continuous processes including DevOps and BizDev (business strategy and development).

• Forsgren, Sabherwal, and Durcikova. European Journal of Information Systems¹⁰

Highlights the roles adopted when sharing and sourcing knowledge in a system management context, and identifies strategies for optimizing outcomes.

• Forsgren, Durcikova, Clay, and Wang. Communications of the AIS⁵

Identifies the most important system and information characteristics of tools for users who maintain systems.

• Dennis, Samuel, and McNamara. Journal of Information Technology¹

Highlights that maintenance effort should be considered during the design phase and calls this DFM (design for maintenance). The authors present insights into how the links among documents should affect both the maintenance effort and use.

• Sharma and Rai. European Journal of Information Systems¹⁹

Investigates how an organization’s computer-aided software engineering adoption decision is influenced by individual factors of IS leaders and technological factors.

• Shaft and Vessey. Journal of Management Information Systems¹⁸

Aims to examine software maintenance as interlinking comprehension and modification; the relationship between these two factors is moderated by cognitive fit.

Conferences

• Trigg and Bødker. ACM Conference on Computer Supported Cooperative Work²⁰

Examines how people tailor their shared PC environment and presents an understanding of how software development tailoring can be helpful in designing systems that better fit the demands.

• Lwakatare et al. Hawaii International Conference on System Sciences¹⁵

Investigates key challenges of DevOps adoption in embedded system domains.

• Wiedemann and Wiesche. European Conference on Information Systems²²

Presents an ideal skill set that DevOps team members should adopt to manage the software delivery life cycle.

Back to Top

Practical Challenges and Opportunities

Implementing DevOps presents several challenges. First, technology—and the resulting organizational transformation—is difficult, but strong leadership can help. As many practitioners note, managing and enabling the cultural changes inside an organization can be a more difficult and important challenge than implementing the technical changes. Good leadership is vital. One approach studied in several contexts, including DevOps, is transformational leadership; this style uses five dimensions (vision, intellectual stimulation, inspirational communication, supportive leadership, and personal recognition) to inspire and guide teams.

Second, DevOps requires a custom solution for each organization. Each context is unique, and a prescriptive approach to DevOps implementation and adoption is unlikely to be successful. Teams and organizations pose a unique set of challenges and cultural norms. Each one should adopt and adapt its own approach to achieve Dev Ops success. The adaptation of Dev Ops should include development of not only technical, but also cultural, process, and measurement practices. The work of creating a unique and seemingly ad hoc technology transformation journey is difficult and may be daunting, so many organizations look for step-by-step guides. However, these are not likely to provide solutions (beyond basic advice such as “automate your toolchain”) and are usually offered by those trying to sell you something.

Third, each DevOps solution should encompass a holistic view, consisting of automation (including tools and architecture), process, and culture. In many traditional approaches, specialist knowledge in one area (for example, development) is leveraged to accomplish a task before passing it off to another group. In DevOps, a move from this high specialization to include a broad understanding of more areas is necessary. (Some call this T-shaped knowledge, with the top part of the “T” representing broad knowledge, while the stem of the T represents deep understanding in one area of expertise.)

This allows people to understand how their work will affect and interact with more areas of the technical stack; this often requires significant additional learning and responsibilities in the transition to DevOps. Organizations should provide training and education, and not just expect technologists to augment their learning independently. Note that while some technologists consider this expansion of responsibilities and knowledge exciting, others may push back, especially those who are just a few years from retirement and comfortable in their work roles, or who see sharing information about their roles as a risk to job security. Organizations must consider these training and cultural challenges in particular and respond accordingly. (As already noted, DevOps is about much more than just technology!)

Back to Top

Conclusion

DevOps is about providing guidelines for faster time to market of new software features and achieving a higher level of stability. Implementing cross-functional, product-oriented teams helps bridge the gaps between software development and operations. By ensuring their transformations include all of the principles outlined in CALMS, teams can achieve superior performance and deliver value to their organizations. DevOps is often challenging, but stories from across the industry show that many organizations have already overcome the early hurdles and plan to continue their progress, citing the value to their organizations and the benefits to their engineers.

Back to Top

Acknowledgments

The authors would like to thank our anonymous reviewers, whose guidance and thoughtful feedback helped us to shape and refine this article.

Back to Top

Back to Top