New software features and updates are the lifeblood of many organizations today, so the faster they can bring them out, the more competitive and customer-responsive they will be. One of the techniques progressive organizations are using to achieve that goal is release automation. Understanding what goes into release automation is key to realizing its benefits.
It’s easy to think of release automation as going the last mile into production, but automation throughout the dev/test/deploy cycle is critical to true release automation. Moving code to those dev/test environments, including performance and acceptance testing, as well as deploying feature branches and deploying changes to the infrastructure or the database, can all be automated.
“For many companies, production is the environment they deploy to less often; a team might deploy dozens of times a day to dev/test environments, and only once or twice a week to production,” said Paul Stovell, founder and CEO of release automation software provider Octopus Deploy. But, he cautioned, the deployments to production are the ones that carry the most risk.
RELATED CONTENT:
How these tool providers address the issue of release automation
A guide to release automation tools
The goal to narrow that risk is to share “as much as possible between your pre-production and production deployments, so that each time you deploy to dev or test, you are gaining confidence that your production deployment process will work,” he said.
The complexity of modern apps
Deploying and releasing software was more straightforward, albeit slower, back when the waterfall development method ruled the world. It was just one big deployment, fully tested and ready to go. But microservices and containers have added a great deal of complexity to deployment.
Mike Zorn, a software engineer at feature management platform provider LaunchDarkly, said, “Deploying one application through a fleet of servers is not a trivial thing. Microservices have really magnified deployment complexity, because now you have different versions being shipped on different schedules. And they depend on one another. So, even if your deployment process is highly automated, you need to have some kind of ability to identify when it’s safe to turn something on. You might need to make sure this version of the microservice is running before you can release a feature on a different microservice, because it depends on that thing being present.” Tracking those dependencies, he said, can be “a real nightmare.”
“In my mind,” he added, “release automation really is about establishing a repeatable release process that has the right decision-makers involved, and presents them with the data they need to make that decision on whether or not to release something.”
Octopus Deploy’s Stovell describes two aspects of release automation: the high-level orchestration of the release process, and that last mile into production. The orchestration effort should involve modeling the way a release progresses between environments, which steps in the process can be skipped, environment configuration, promotion rules, and communication around all of that. The last-mile piece, Stovell said, more simply involves “how the bits make their way from the release automation server or artifact repository onto the running production server.”
Under this definition, microservices don’t really change the last-mile deployment, but they add a lot of complexity to the orchestration aspect. “I like to say the microservices moves inter-team communication from a compile-time problem to a runtime problem,” Stovell said. “By this, I mean with a monolithic architecture, deployment (runtime) is very straightforward, but the downside is that developers have to talk to each other a lot more while they write the code — compile time.” Meanwhile, microservices and containers — particularly platforms like Kubernetes — help a lot with the runtime deployment, but don’t help much with the orchestration end, he noted.
Mitigating risk
With this added complexity, it’s important to be able to respond quickly if a release into production didn’t function as planned or created a vulnerability. The use of canary releases, blue/green deployments and feature flags are effective methods to reduce risk.
Because of the nature of microservice dependencies, LaunchDarkly’s Zorn noted that if you deploy the services in the wrong order, with one going live before another dependency, “you could have created a pretty nasty thing.” Being able to turn off the feature can be beneficial in that architecture, he said. “You can just roll it back to the last successful release that you had, and figure out and wait until the dependencies all catch up with each other, and then you can re-release,” Zorn explained.
Because of its access to production servers, the release automation system has much more fine-grained access control and auditing requirements, Stovell said.
And LaunchDarkly’s Zorn added, “Getting stuff on the servers, getting code tested and all that, that’s just pretty well established but then the actual turning the feature on for the customers is where there’s a large degree of control that you can have that is often not capitalized upon.”
Making the decisions
To many, automation is a double-edged sword. While there are some big benefits to release automation — creating repeatable processes, taking mundane tasks away so engineers can work on higher-value projects — many organizations still want people and not machines to make the final go/no-go decisions.
Zorn explained: “Today, you can say, ‘Okay, I’m ready to roll this feature out.’ But I want to do it in a staggered way. So, let’s do like 10% on Monday, 1% on Tuesday, 50% on Wednesday, and so on, for like, three weeks or something. And then each day, you’re going to be checking the metrics, and monitoring things that are important to you to make sure that it’s safe to proceed. In that release, that’s a procedure that I’ve personally gone through and done a number of times. But one thing that I’m looking toward is a world where the tool that is executing that schedule, adds the ability to look at those metrics on its own, and make a decision like, ‘Oh, this page load time is starting to deteriorate with the people that are getting a new feature, let’s halt the release automatically.’ That kind of thing is a solid use case. In these cases, the computer is in a better position to decide.”
But the ability for these automation tools to make decisions is still fairly new and niche. “We have an engineer that came from Twitter recently, and they had a mechanism that could do some of this stuff. But that’s the only company that I’m aware of that is really doing stuff like that. So it’s pretty advanced. But it is something that a lot of people maybe want. Their dream is that an engineer can go on vacation, while their features are rolled out. And if something goes bad, it just automatically stops. You don’t have to think about when people are taking vacation when you’re shipping stuff.”
Octopus Deploy’s Stovell said the ability for machines to make the call on releases depends upon the maturity of the monitoring capabilities of the organization, rather than the tool itself.
For instance, a company with a single product — say a SaaS web app — with hundreds of engineers working on it, can invest in building a very sophisticated, high-velocity pipeline that is uniquely tailored to that project, he explained. “The release automation part is actually easy; the hard part is the sophisticated ops/monitoring setup to alert them to problems and roll back bad releases automatically. Such a company would want to optimize for the velocity of change and time to market, and would need to live with some of the architectural constraints that Continuous Deployments/automated rollbacks require, for example database changes being N-1 backwards compatible, and a high percentage of automated test coverage.” For a team like that, he said, “it makes sense to let the machine make the call.”
He next offered the example of a similarly sized company with a large portfolio of products — say, an insurance firm — with many of those built on different technology stacks at different times. Some projects in the portfolio might be under active development, some may just have the occasional bug fix here or there. As a result, Stovell said, that company is “unlikely to have the same ability to invest in the monitoring/ops maturity needed, because there’s such a wide range of applications and infrastructure. And since changes to some projects are less frequent, velocity of change from dev to production might come second to other concerns, like keeping things simple and consistent between teams and projects. So for a company like that, it might make a lot more sense for there to be a person making the final call that a release is ready to go to production.”
What will it take to achieve continuous deployments to production without human involvement?
Before a team is ready for continuous deployments to production without human involvement, Octopus Deploy founder and CEO Paul Stovell said he would expect to see:
- Zero downtime deployments — release automation tools can help with this, but it also requires architectural and infrastructure changes.
- A high level of automated test coverage — unit, integration and automated UI tests. This is not a release automation issue.
- Sophisticated monitoring and alerting that can detect when a release is bad. There are plenty of monitoring tools available, but “out of the box” they are all noisy and can lead to false positives. It requires significant and ongoing time investment from operations teams to detect the right errors, ignore the wrong errors, and connect that feedback loop to the release automation process.
CI/CD vs. release automation
When developers think about release automation, Octopus’ Stovell said they probably assume their CI server should handle it. “It’s really the only team-level automation tool that most developers are familiar with.” But, he pointed out, there are fundamentally distinct problems between continuous integration and releasing the code.
“A common setup we see is a software team using their CI/CD server to attempt to automate every part of the software delivery process, and then operations teams using different automation tools to automate all of the infrastructure and runbook automation,” he said. “To me, that’s a silo that doesn’t make any sense. I think it’s part of the historical evolution — developers deciding to automate deployments, and using their CI system to do it.” So, riffing on a popular expression, Stovell added, “When all you have is a Jenkins-shaped hammer, every automation task looks like a Jenkins job.”
A good CI system, he explained, needs to know about things like source control, branches and unit testing. A good CI process is designed to fail fast — “There’s no point trying to run unit tests if the compile failed, and no point compiling if resolving dependencies failed,” he said — so the process is optimized to be a fast feedback loop to developers.
On the other hand, a good release automation system has a completely different problem space to think about, Stovell continued. It needs to model all of the environments a team deploys to, and needs patterns like blue/green, canary, and rolling deployments. And failing fast is not the goal: “If you have 20 front-end web servers and one can’t be deployed to, you can’t fail the deployment. Remove it from the load balancer and keep going, then deal with it after.”
These problem areas that release automation tools have to worry about is the same problem areas that IT operations automation tools need to think about, he said. “They both need to model environments. They both need production secrets. They both need to be highly secure and have solid access control and auditing,” he said. “Release automation interacts with load balancers to add/remove servers during a deployment; operations automation interacts with load balancers to add/remove servers to install system updates. There’s a lot of commonality, but the evolution of the tooling stacks means companies often use different tools.”
And it’s the use of these different tool sets that maintains silos that organizations say they need to break down. “We talk about DevOps a lot – we gather all the developers, and all the ops people, into a room and we hold a meeting, and we talk about how we’re going to collaborate more,” Stovell said. “And then both groups go back to their desks, and use two completely different tool sets, to automate the same types of tasks. That makes no sense to me. It’s the reason we introduced Runbooks to Octopus Deploy recently — as far as we know, Octopus Deploy is the first release automation tool to explicitly model both Release Automation and Runbook Automation in the same tool.”
