The long-distance runner rounds the track and starts down the final 200-meter straightaway, but stumbles dramatically. The back of the pack has nothing left to give. The leader lopes ahead effortlessly. Meanwhile, a surprise surge propels an athlete past four frontrunners. She slides across the line in time to win the silver medal.
The Olympics are an apt metaphor for peak performance of software deployment and release. Runners spend years circling the track, becoming ever more efficient at their sport. They work their way up through regional, national and world competitions, practicing delivery under pressure. But only the best make it to the Olympics, and once there, some fail ignominiously while others show that strategy is crucial when it comes to the final heat.
Avoiding dramatic blowouts in the “last mile” of the software life cycle has gained popularity thanks to David Farley and Jez Humble’s book “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.” Several new tools focus on elements of the life cycle after development and testing. And in 2012, the focus is increasingly on deployment, as open-source continuous integration tools such as Jenkins gain popularity.
Even the distinction between deployment and delivery is being actively debated, along with the utility of canary releases and feature toggles. These are all signs that the DevOps movement—bringing operational concerns into the early parts of the requirements and development cycle—is gaining mainstream awareness.
But like the weekend warrior whose training plan falls far short of competitive standards, is the average software team even ready to dream of continuous delivery? Has its prerequisite—continuous integration and build management—been met?
What continuous integration isn’t
A recent report by analysis firm Voke on agile that found only 49% of enterprises were using continuous integration. What’s worse, far too many teams are doing it wrong:
• It’s not just running a tool. “When people say they do continuous integration, what they actually do is they’re running a continuous integration tool, and that’s not the same as doing continuous integration. Crucially, continuous integration is a practice, not a tool,” said Humble. And what is that practice? “The essence of it is, you’re always pushing into mainline, into trunk, and working off trunk.”
• It’s not nightly builds. Development teams shouldn’t deceive themselves, Humble said: “A nightly build is not continuous integration. If you are not checking into trunk at least once a day, if you don’t have a suite of tests that runs in a few minutes, if you don’t pay attention when the build goes red and make it the highest priority of the team to fix it now, you’re not doing continuous integration.”
• It’s not hours of tests. Even the most extreme continuous deployment (see Timothy Fitz’s report, “Doing the Impossible 50 Times a Day”) depends on a test suite that takes just nine minutes (albeit distributed across 40 machines). At the other end of the spectrum, too many companies have no comprehensive automated testing in place.
• It’s not sifting through failure reports. “Friends don’t let friends grep log files,” said Anders Wallgren, CTO of Electric Cloud. “If you’re running a two-hour test suite and having to carefully watch the output, you don’t have to. We’re watching those builds in real time and can pinpoint what the problem is for you.”
• It’s not just running build and unit tests on a feature branch. According to Humble, continuous integration only applies to regular merges into the centralized code repository, or trunk, a subject of some controversy among developers fervently embracing distributed version control systems such as Git.
As support for his argument that continuous integration needs not require a special tool, Humble points to a 2006 article by James Shore, “Continuous Integration on a Dollar a Day,” which suggests all you need is an old workstation (to run the builds), a rubber chicken (to take to your desk to alert everyone that you have checked in changes and a build is running), and a desk bell (which you ring if everything passes).
#!
Build management to deployment
It’s more likely, however, that you are using a tool to automate the integration and build processes. Whether it’s JetBrains’ TeamCity for .NET developers, Atlassian’s Bamboo or Apache Maven for Java developers, CruiseControl.rb for Ruby developers, or Jenkins (née Hudson) for any language, there’s something at every price point. While Jenkins is free and open source, there’s ThoughtWorks’ Go in the upper mid-range. UrbanCode’s uBuild and Electric Cloud’s ElectricAccelerator and ElectricCommander are at the high, six- to seven-figure end.
Further, build management and continuous integration tools have become increasingly indistinct: “There is enough overlap between the tooling that it was easy for CI tools to steal build management features, and vice versa,” observed Eric Minick, technical evangelist for UrbanCode. “In 2001, we released the Anthill open-source build management tool, which is long since dead. Within a month, CruiseControl came out as a continuous integration tool. I don’t think we and the CruiseControl guys saw us as being in the same space. But the net result was that the tools learned from each other.”
But there are crucial differences between the two activities. At heart, only developers care about continuous integration. Build management tools go further, creating standalone runtime artifacts that are traceable back to requirements and ready for handoff to QA or deployment. In fact, the extra information created by a build tool adds unnecessary overhead to continuous integration.
Build tools may be task-oriented (Ant, NAnt and MSBuild) or product-oriented (Make), which has repercussions in terms of speed and success. The product-oriented tools generate a series of files and execute them according to their timestamp order. “This is great when you’re compiling C or C++ (for example), because Make will ensure that you only compile those source-code files that have changed since the last time the build was run. This feature, known as an incremental build, can save hours over a clean build on large projects,” wrote Farley and Humble in “Continuous Delivery.”
There are additional details around optimizing builds for managed code and other platforms, but assuming you’ve got the right tool for the job, the slippery slope starts here. After the initial implementation of a check-in to version control launching a build loop and unit tests, it becomes attractive to stress-test the nightly build for performance.
“What happens as you follow this path is you end up automating your build, storing it somewhere, automating deployments, running tests,” said Minick. “Now you’re sitting with a build that’s been tested and has new features. You know how to do deployments well, so it would be trivial to put it out to production. Why not put it out to the last step?”
“A lot of people don’t understand if you have a build problem, you also have a deploy problem,” said Wallgren. Using Jenkins, for example, may be fine for a small team, but “Jenkins has limitations. It’s not an enterprise software product.” Jenkins does not scale in ways that work for large companies: “If you have 1,000 developers, if you have 12 different products being worked on simultaneously, if you want to do things like preflight build, you can’t use Jenkins.”
Finally, looking out toward deployment, new efficiencies become critical. According to Humble, it’s important to “build it once, then promote to binary, rather than rebuilding from source all the time when you deploy. Build the package once and use that package all the way through the rest of your deployment pipeline.”
The new deployment market
Even if continuous integration and build management haven’t achieved no-brainer status quite yet, the tools market is eagerly looking ahead to deployment as the next sweet spot. In addition to uBuild, UrbanCode now offers uDeploy to establish repeatable, error-free, scalable application deployment processes, and provide visibility into what version is where. The tool has newly added security controls via Environment Gating, which allows users to enforce a particular promotion path, or status, for artifacts.
For example, a status might require that in order for an environment to push to production, it must first have passed system integration and user acceptance testing. There is also additional security around who is allowed to apply a status and permit a version into production.
Electric Cloud has jumped into the deployment fray even more recently, launching ElectricDeploy in 2012 on top of its ElectricCommander build platform. The tool uses a model-driven approach to reduce the variability of deployments across all environments: Dev, QA, preproduction and production. “Surfacing failure,” as Wallgren put it, is something the tool does well with features termed code-safe for runtime troubleshooting, run-safe for failure definition, and recover-safe so deployment teams can retry and recover quickly.
“We surveyed our customers about the number of times that an app is deployed during the life cycle, and the result was around 10,” according to Wallgren. Doing that many ad hoc deployments of the same application is clearly inefficient, he said. Instead, ElectricDeploy lets you model the process by dragging out the tiers, including logical ones for setup, teardown, monitoring, defining databases, and configuring firewalls and load balancers, among other elements.
“By the time you get to user acceptance testing or preproduction, the rest of those tiers start to show up,” said Wallgren. “As you go through the life cycle, you’re developing and debugging your deployment process. Once you have a process that works for you, you save it, and nothing can change it.” Like other tools in this space, ElectricDeploy’s dashboard gives easy visibility into what’s in the deployment pipeline.
Taking a slightly different perspective is Nolio, which offers the Nolio Application Release Operations Suite, containing Nolio Release Operations Center for enabling multiple continuous releases throughout the life cycle; Nolio ASAP Release Automation v4, a new version of the company’s flagship application-deployment, maintenance, remediation and recovery tool with “zero-touch deployment”; and Nolio ASAP Cloud Pack, designed for launching apps in—you guessed it—private, hybrid and public clouds.
“We come from the deployment and production side, not from the build side,” said Jacob Ukelson, Nolio’s vice president of strategy. “That’s where we grew up. We’re starting to talk about release operations, because deployment is only a piece of what’s going on. The world has changed in a few ways. Users have mobile devices and are expecting apps to be available all the time. Infrastructure is becoming more elastic and agile with virtualization and the cloud.”
These new pressures mean companies must be able to conduct corrective and preventive operations. “There are all sorts of maintenance and remediation passes to ensure a production release provides value to users,” said Ukelson. Activities along this deployment pipeline or promotion path are what Nolio calls release operations, and typical problems may occur around heterogeneous geographies or devices. “We certainly hook in to various build tools like Jenkins and Team Foundation Server so you can trigger us when you build.”
However, Nolio claims to be more than a commercial version of Chef or Puppet, which are open-source tools for IT automation. “There’s talk about moving up the stack for Puppet and Chef into this orchestration layer,” Ukelson noted.
Finally, in the .NET space, there’s a brand-new tool: Octopus Deploy. Launched in 2012 by a lone, London-based Australian developer named Paul Stovell, Octopus is an automated deployment system designed to fill the gap between builds and delivery. “In Australia, bandwidth is limited. You use data centers, but you can only access them through remote desktop. How do you take a bunch of code and push it out to a bunch of machines without using FTP or file shares over a VPN?” he explained.
While Stovell often saw good use of version control and build management, “Deployment is patchy. You have some guy with a beard writing a bunch of batch files,” he said. The fact that deployment seemed to bedevil so many companies where he did consulting work led to his solution, which is written in C#, communicates via WCF, runs PowerShell scripts, and uses a packaging format similar to ZIP files called NuGet, which is popular for building .NET components.
Octopus has an agent for production machines, called, aptly, a tentacle. “It listens on a port that you configure, and every communication is encrypted using pairs of certificates,” said Stovell. “The tentacle will only accept commands from Octopus, and security (which can be onerous to manage, what with federation, matching user accounts and the like) was a major focus for the tool, along with productivity.
“If you had Octopus installed, taking a Web app building already in TeamCity to automatically deployed would take no more than a few hours. Once deployed to the staging environment, you click one button to promote to the production environment. And the nice thing is you set up a template to say this is how you deploy this product.”
Thanks to their respective histories, Electric Cloud has a larger ISV customer base, while UrbanCode’s is more enterprise IT, but both come from the developer mindset and claim to offer end-to-end solutions from build to deploy. Nolio is more established in the operations space, while Octopus is reaching for .NET shops. None of these tools, however, seem to pique the primary interest of developers, according to the vendors; they’re for DevOps.
“You have two kinds of calls,” said Minick. “One is from the Ops side of the house, and they say, ‘You guys do deployment right? Cool, can I see a demo of your product?’ We start by showing the developer checking in code, then the build, unit tests… By the time we get to deployment, they’ve fallen asleep. Then they go buy BladeLogic.”
Hence the deploy-only tool, which works with any existing build solutions, without the soporific effect on DevOps. “When someone wants an end-to-end solution, that’s a development or release manager wanting to deploy out,” he said.
Feature toggles and canary releases
Beyond the automation piece, however, there’s even more that can be done with continuous delivery.
There are numerous terms for it: Dark features, feature toggles, A/B releases, or canary releases. The idea is the same: Deploy functionality out to the customers, but only switch it on for some of them. Facebook is known for this approach. Though some argue this adds needless complexity to releases, according to Humble, there are three main use cases for feature toggles: “One, it’s a cheap way to roll back. If you roll out software and it’s broken, you just turn it off. Another important use is managing unexpected load on your system. Facebook let you choose your username on a certain day and time, and it turned off some features to cope with the increased load. The third use is doing testing on the features that you’re building.”
Accounting for all your customizations, plus new features, is a software engineering problem that’s been around for a long time, observed Randy DeFauw, technical marketing manager for Perforce Software. While some “simply put a bunch of flags and switches in source code and make files, that doesn’t scale, and you end up with spaghetti code. Or you do it all in SCM with branching. That demands that you have a lot of discipline: You have to invest in your release-management team to make that work for you.”
But the most promising approach is to add yet another layer of dedicated tools, such as what’s being done with electronic design automation tools. “The two industries I see it in most are EDA [electronic design automation] and video games. Gamers may have a core engine that they tweak with different front ends for different devices,” DeFauw said.
For an example of precisely how even the most complex implementations are leveraging continuous delivery and deploying mixed feature sets, Humble points to an upcoming book on how HP used large-scale agile to transform LaserJet FutureSmart firmware by Gary Gruver, Mike Young and Pat Fulghum.
But the canary release is only for those prepared to lose a canary or two. Most companies aren’t there yet. “Some of our SaaS companies are doing this. Our enterprise customers are talking about it, but not doing it. This is really forward-thinking,” said Ukelson.
