I’ve spent a lot of time looking at how organizations are using DevOps to improve their software delivery cycle time, their ability to innovate, and their ability to improve quality. I’ve heard some people go so far as to say that DevOps has replaced Agile, but I don’t think that’s true. If anything, DevOps reinforces and supports Agile, and Agile is still at the heart of DevOps. The essential features of Agile - working in small increments (batches), delivering working software frequently based on business priorities, working in small self-managing cross-functional teams, and measuring progress by working software - are essential for DevOps to work. In my view DevOps makes explicit what many Agile teams had already learned but may not have formalized. DevOps also spreads Agile principles to other parts of the organization, especially Ops. In truth, Agile and DevOps are symbiotically and inextricably linked.
The Agile Manifesto says Agile teams value working software over comprehensive documentation. This means that they need to build and test software. A lot. All of the time, in fact. Ideally, they need to build every time they commit a change to the source code repository. But builds are just the starting point - they also need to test every time they make a change. Not just unit tests, but functional tests, regressions tests, performance and scalability tests, all driven from APIs. And not just in simple environments, but ones that look as much like production as possible. When they do this they deliver better code.
To do this, they need to practice trunk-based development. They need continuous integration automation that drives not only the build process but a robust API-based testing process, every time code is changed. They also need to have robust test environments available on demand, which means using infrastructure as code or other environment provisioning automation. And at some point, when they are fully responsible for supporting the application in production, they need release automation tools to reliably deploy code. Some people have taken the Manifesto’s statement that Agile teams value Individuals and interactions over processes and tools to mean that tools aren’t important. People are most important because they are the ones who actually create software, but in practical terms those people need tools.
Demonstrating working code in a demo at the end of every sprint is essential. Demos are not the same as the real world, however. Demos don’t test real workloads in real-life threat environments. Demos don’t have to play nice with other applications. Demos don’t have to be fault tolerant or self-healing. Demos in a test environment are necessary but not sufficient. Scalability, security, reliability, and maintainability are essential. Demos can focus too much on the look and feel of the application and not enough on its real-world characteristics. Agile teams who support their applications in real world environments learn this in a hurry.
Realistic production-like testing environments are a good first step. Advanced deployment techniques like blue-green deployments and canary deployments, deployments to subsets of the user community, provide insights that demos never can. Testing the deployment processes themselves by frequently deploying to real-world environments helps to remove inconsistencies that are significant sources of production incidents. Agile teams need to adjust their definition of done to eventually encompass deploying to real customers - until they do they may be fooling themselves about the quality and value they think they are delivering.
The Product Owner is the single authority on what needs to be built and the arbiter of “done”, at least until real customers speak. Product Owners, even great ones, are people like the rest of us, with their own biases and blind spots. Adding other stakeholders to demos can help by expanding perspectives, but they, too, have blind spots. The ultimate arbiter of value is the customer, and the faster a team can deliver to real customers, the faster they will confirm or reject the theories they have about what those customers really need. This is not a criticism of the Product Owner’s role, or the opinions of stakeholders, merely a reflection of reality.
Performance and scalability matters. Security matters. Services, microservices, and APIs especially matter. The more loosely coupled an application, the easier it is to change one part without that change breaking lots of other parts. APIs can be regression tested as part of the Continuous Integration process. Minimizing code dependencies also has a pay-off when tackling larger applications: coordinating work with other teams through shared APIs is a lot easier than sharing data structures. A lot of complex program management and team coordination overhead can be avoided when team dependencies are mediated through APIs. Techniques like service virtualization (otherwise known as mocking and stubbing) can be used to simulate commitments while the other team works on delivering the new API.
One of the reasons that legacy application code is so hard to release frequently is the the applications tend to be monolithic, sharing data structures and databases with other applications. They are hard to change because every change has unknown side effects. Getting this code back under control will take years, and it may simply be easier to rewrite it. But modern applications don’t have to be this way, and Agile teams need be work hard to see their applications loosely coupled, lest these applications become the legacy applications of tomorrow.
The smaller the release, the fewer the dependencies. The fewer the dependencies, the fewer things there are to go wrong. If you want to reduce the risk of a release, reduce the size, don’t double-down on control processes that actually increase release size by creating a process so onerous that people will do anything to avoid it, like releasing less frequently. Smaller releases makes coordination easier. They make user adoption easier because there are fewer changes or new features to learn. Smaller more targeted releases make it easier to evaluate their impact because their are fewer things to measure. This makes feedback faster, which reduces waste and helps teams to better scope future releases. Smaller, more frequent releases are good.
They are good, but only if a team is continuous building and testing their software. They are good only if release processes are simple, streamlined, and predictable. They are good only if applications are loosely coupled and don’t create a cascading series of problems when something fails.
DevOps practices help Agile team to automate the mundane and routine, letting their focus on writing high quality modular, loosely coupled code that delivers the right solution. DevOps needs Agile’s business-driven (through the Product Owner) backlog and pipeline management capabilities. It needs the Agile teaming model to create high-performing cross-functional teams. Agile teams need to add Ops and Security skills to their repertoire as they increasingly take on responsibility for managing applications in production.
Agile and DevOps are not really separate things. Agile has always embraced excellence in engineering principles. Having to deliver and support applications in production expands that focus. Doing so at significantly faster cycles means that automation is now an even more important part of those engineering practices.
Agile needs automation to really boost velocity
The Agile Manifesto says Agile teams value working software over comprehensive documentation. This means that they need to build and test software. A lot. All of the time, in fact. Ideally, they need to build every time they commit a change to the source code repository. But builds are just the starting point - they also need to test every time they make a change. Not just unit tests, but functional tests, regressions tests, performance and scalability tests, all driven from APIs. And not just in simple environments, but ones that look as much like production as possible. When they do this they deliver better code.
To do this, they need to practice trunk-based development. They need continuous integration automation that drives not only the build process but a robust API-based testing process, every time code is changed. They also need to have robust test environments available on demand, which means using infrastructure as code or other environment provisioning automation. And at some point, when they are fully responsible for supporting the application in production, they need release automation tools to reliably deploy code. Some people have taken the Manifesto’s statement that Agile teams value Individuals and interactions over processes and tools to mean that tools aren’t important. People are most important because they are the ones who actually create software, but in practical terms those people need tools.
The gap between “potentially releasable” and “actually released” can be vast
Demonstrating working code in a demo at the end of every sprint is essential. Demos are not the same as the real world, however. Demos don’t test real workloads in real-life threat environments. Demos don’t have to play nice with other applications. Demos don’t have to be fault tolerant or self-healing. Demos in a test environment are necessary but not sufficient. Scalability, security, reliability, and maintainability are essential. Demos can focus too much on the look and feel of the application and not enough on its real-world characteristics. Agile teams who support their applications in real world environments learn this in a hurry.
Realistic production-like testing environments are a good first step. Advanced deployment techniques like blue-green deployments and canary deployments, deployments to subsets of the user community, provide insights that demos never can. Testing the deployment processes themselves by frequently deploying to real-world environments helps to remove inconsistencies that are significant sources of production incidents. Agile teams need to adjust their definition of done to eventually encompass deploying to real customers - until they do they may be fooling themselves about the quality and value they think they are delivering.
Product Owner and stakeholder feedback is great but real customer feedback is the true test of value delivered
The Product Owner is the single authority on what needs to be built and the arbiter of “done”, at least until real customers speak. Product Owners, even great ones, are people like the rest of us, with their own biases and blind spots. Adding other stakeholders to demos can help by expanding perspectives, but they, too, have blind spots. The ultimate arbiter of value is the customer, and the faster a team can deliver to real customers, the faster they will confirm or reject the theories they have about what those customers really need. This is not a criticism of the Product Owner’s role, or the opinions of stakeholders, merely a reflection of reality.
Application architecture matters
Performance and scalability matters. Security matters. Services, microservices, and APIs especially matter. The more loosely coupled an application, the easier it is to change one part without that change breaking lots of other parts. APIs can be regression tested as part of the Continuous Integration process. Minimizing code dependencies also has a pay-off when tackling larger applications: coordinating work with other teams through shared APIs is a lot easier than sharing data structures. A lot of complex program management and team coordination overhead can be avoided when team dependencies are mediated through APIs. Techniques like service virtualization (otherwise known as mocking and stubbing) can be used to simulate commitments while the other team works on delivering the new API.
One of the reasons that legacy application code is so hard to release frequently is the the applications tend to be monolithic, sharing data structures and databases with other applications. They are hard to change because every change has unknown side effects. Getting this code back under control will take years, and it may simply be easier to rewrite it. But modern applications don’t have to be this way, and Agile teams need be work hard to see their applications loosely coupled, lest these applications become the legacy applications of tomorrow.
Small batch size is really important for fast feedback cycles and reducing release risk
The smaller the release, the fewer the dependencies. The fewer the dependencies, the fewer things there are to go wrong. If you want to reduce the risk of a release, reduce the size, don’t double-down on control processes that actually increase release size by creating a process so onerous that people will do anything to avoid it, like releasing less frequently. Smaller releases makes coordination easier. They make user adoption easier because there are fewer changes or new features to learn. Smaller more targeted releases make it easier to evaluate their impact because their are fewer things to measure. This makes feedback faster, which reduces waste and helps teams to better scope future releases. Smaller, more frequent releases are good.
They are good, but only if a team is continuous building and testing their software. They are good only if release processes are simple, streamlined, and predictable. They are good only if applications are loosely coupled and don’t create a cascading series of problems when something fails.
Agile teams need DevOps practices, and DevOps needs Agile’s teaming model and connection to business value to drive better results
DevOps practices help Agile team to automate the mundane and routine, letting their focus on writing high quality modular, loosely coupled code that delivers the right solution. DevOps needs Agile’s business-driven (through the Product Owner) backlog and pipeline management capabilities. It needs the Agile teaming model to create high-performing cross-functional teams. Agile teams need to add Ops and Security skills to their repertoire as they increasingly take on responsibility for managing applications in production.
Agile and DevOps are not really separate things. Agile has always embraced excellence in engineering principles. Having to deliver and support applications in production expands that focus. Doing so at significantly faster cycles means that automation is now an even more important part of those engineering practices.