To many, release automation is that last stage of the DevOps pipeline, when software is made public. Yet, to many, release automation remains largely misunderstood and so is not as widely utilized in organizations that claim to be agile and doing DevOps.
“I don’t know if people are recognizing the benefits of application release automation just yet,” said Tracy Ragan, CEO and co-founder of OpenMake, which offers automation from build to release.
One person who does understand the benefits of application release automation is Tom Sherman, divisional vice president of DevOps at health insurance management company HCSC. “In mid-2016, we started a project to move all of our applications onto a release automation platform. We’ve been able to move about 400 applications to implement a release automation process for them.
We have automated the process of building the application and then doing the deployments. We’re seeing a lot of benefits in doing that in terms of speed and resources saved, as they don’t have to work like they had to do in a manual release process, particularly on the infrastructure side — the teams that used to have to support us by loading our applications out onto servers and so forth.”
Bob Evans, director of applications engineering at email delivery service SparkPost, sees the benefit of getting new features out to customers more quickly. “This is crucial because it allows an organization to get feedback during the development cycle and iterate accordingly,” he said, adding “I don’t believe you should take operations out of the equation, it’s just their responsibilities shift. DevOps puts more ownership and success of the overall service on the team. It also empowers team members to address issues before adding more features.”
Automation is used in many areas of IT, from test and build automation on the developer side to automating steps in IT operations. Yet some fear remains. As Electric Cloud’s CTO Anders Wallgren noted, “Release automation managers have a bit of ‘we’re different. We’re a duck-billed platypus, you can’t possibly automate what we do. It can’t possibly work for us, we’re different.’ All of the usual kinds of phrases come out when you start to dig in at particular organizations. There’s a little bit of fear there for this kind of stuff. We like other companies have had to work through that a little bit.”
Not only a fear of losing their jobs, but a fear of turning over the keys to a machine that in their minds could lead to releasing broken or bug-filled software. Release managers, Wallgren said, should spend more time talking with the people who’ve created the functionality upstream and downstream. “They still seem to not really be getting it yet. They say, ‘Look, if we automate everything, what if something goes wrong, are we going to release software that doesn’t work?’ All the usual reasonable fears that people have.”
And who are these release managers? “I have seen several instances where people that work in the release management arena are a little bit divorced from the details of what it takes to get a release out the door,” Wallgren said. “What I mean by that is often times, they’re project managers, and I don’t mean that in a perjorative way; they’re there. Their job is to herd the cats. We all know how much fun that is. Sometimes, the notion of automation to them is a little bit scary, for a number of reasons. Sometimes it maybe dives into the technical areas a little too much, and a kind of nervousness about, well, am I going to be necessary if we automate everything. That silo is starting to learn about this and figure out what it means to them.”
They’re also not necessarily exposed to the innards of the releases, he said. “Some release managers are more like product owners; they understand what’s coming in the release, what’s in there, why it’s in there, they have their finger on the pulse. Other release managers are like, ‘I wait until people tell me what to do and then I rubber-stamp it and it goes.’ I’m probably exaggerating on that, but there is a bit of that.”
To script, or not to script
In most organizations today doing continuous integration and deployment (CI/CD) — the precursor to release automation — the use of scripts remains prevalent. Some, though, see the use of scripts as hampering CI/CD due to scripts being static in nature and unable to adapt across pipelines. “We all have a waterfall approach to CD; scripts come from waterfall,” OpenMake’s Ragan said. “It is still built around states. In dev, test and production, it’s a very fast waterfall approach, but it’s still waterfall. When the build runs, kick off tests. If the tests pass, go to production.”
She recalled a time before the overhyped need for speed when organizations had the luxury of time to tweak scripts to adapt to different environments. Today, she said, “people need repeatable deployment processes from dev to test to production very quickly.” On the speed issue, she opined, “Companies need the choice to be fast or not. What they all need, though, is to be responsive to their market.”
HCSC’s Sherman said repeatable processes provide reliability, but believes scripts have remained valuable. “Anything that you script, you’re executing the process the same way every time you do it, which eliminates some errors that we were seeing in the manual process. When you had different support staff working with you they might not have remembered a step that had to be executed, or the sequence of the steps that had to be executed in a particular order to make it work. The scripting takes that risk away from your process.”
He went on to say that “the way that our process works is that the scripts are created ahead of time and we’re executing the scripts over and over again when we do a deployment. Once you get the scripts created it is an asset that you manage just like any other piece of code, and you version it and test it and implement a new version of it so you’re getting a consistent execution of that script every time.”
Sherman acknowledged that scripts are labor-intensive, to a certain extent, but HCSC looks at tools “that do things more from a configuration standpoint, where you can configure within the tool the way you want to execute your deployment. That’s sort of the next generation of our platform, to take some of the scripting out of the equation and put it into a tool that’s got more ability to have the configuration built in to the execution.
“But those tools are expensive and you’ve got to figure out where you need that level of automation or where the script is adequate,” he continued. “We’re not seeing a lot of volatility in our scripts once we’ve got them set up and they’re working. They’re not requiring a lot of maintenance right now, so we’re not as concerned about that at this point in time. There is a place… we did make a decision on, that we did not want to put into the scripts and we wanted to go to a next-generation tool. So there is definitely a niche for that, and we’re just seeing whether that’s something we want to do on an enterprise level or is it something where we make the tool that does the scripting a part of our business and then go to the next-generation tools. That’s the analysis we’re doing right now.”
All about the pipeline
So, versioning scripts is helpful, but what happens when test, staging and production environments change, as they always do? According to SparkPost’s Evans, “I think building a pipeline that can adapt to changes is key. However sometimes having different load and performance profiles, and hardware constraints makes verifying certain features/fixes tough in different environments.”
OpenMake’s Ragan agreed, saying “to be repeatable, [the flow] has to go across the pipeline. Application release automation allows developers to decide what gets consumed in test and production, and adds security features.”
Yet release automation is “kind of all over the place,” Wallgren said. “Some companies don’t have a release step in their pipeline. They say, if it makes it through the pipeline, it goes straight into release automatically. They don’t have release managers.”
He called release management “a kind of Maginot line. It’s the last defense against releasing things that can hurt them.”
It’s a culture thing
HCSC’s Sherman described the hurdles and roadblocks to implementing release automation.
“It was a difficult project to get started and get through the first couple rounds of it. The other complication I think we had was that our applications, as we went through the analysis, are not all the same, so we had a lot of one-off things that we had to account for, but once we got a process built and an assembly line kind of approach to it, it went pretty smoothly and we really gathered some momentum and were able to do the second half of the project much more quickly than the first half went.
“We had some issues with engagement.. getting the teams to buy into the process. Then we had issues with staffing on my side, of getting the right people who could do the work in place, and then there’s always a problem here of competing priorities. The teams that we were working with had big customer projects that they were working on and we had to fit our work into that.”
Sherman did note that HCSC leadership was committed to making release automation happen as part of a larger organizational digital transformation. “That really helped us turn the corner by having that backing that this was an important project that needed to be done. That really made it work,” he said.
SparkPost has had some form of release automation since it launched in 2015, Evans said. “It was an evolution and over time more pieces of the process were streamlined. We knew in order to be agile and ship features to our customers that release automation was a must. We prioritized critical path and layered in more capabilities as we grew in volume and features. Being an on-premise company before, we had some legacy processes that hindered some of the rollout of the pipeline initially. We also had a new set of challenges as we started rolling out our next-generation architecture in 2017. Our development process changed as well because previously we were working towards quarterly releases. Most features stopped at a test environment until closer to release. Now we release pieces of features to production as they are ready and ship multiple times a day.”
In the world of automation, Wallgren said, “You have to ask the question, ‘what goes wrong today, and how do you deal with that?’ If you can automate it, and it does the same thing every time … then why is that not better? Sometimes you have to push people on that and challenge their assumptions.”
So, when everything is automated, does release management go away in the future? “I don’t think so,” said Wallgren. “But… can questions that release managers want the answers to be populated automatically? Yes.”
