As agile practices continue to gain momentum, and as delivery deadlines grow shorter, the ability to quickly fix bugs and add features is imperative. Despite the hype, not all organizations have adopted Continuous Integration (CI) yet. While continuous builds and CI practices are becoming more common—and a smaller percentage of companies have embraced Continuous Delivery (CD)—there are several technological and cultural obstacles making teams and enterprises less productive than they could be.
“I think we’re still very much in the early phases of what we think of as a complete agile workflow,” said Eric Wittman, general manager of development tools at Atlassian. “It’s not just your practice of assembling a multidisciplinary team and planning shorter increments or shorter work streams and sprints or iterations.”
CI needs more thought
There is no shortage of automated build, CI and CD tools, whether they are open-source projects such as Jenkins and Hudson, or offerings from solution providers such as Atlassian, CloudBees, ElectricCloud, JetBrains, OpenMake, ThoughtWorks and XebiaLabs. The tools need to be supported by processes to be successful, however.
“Most organizations are doing some flavor of CI,” said Mike Mason, head of technology at ThoughtWorks. “Most folks have realized that doing builds on a developer workstation is a bad idea and they need something more repeatable than that.”
Embracing CI means more than just running an integration server, however.
“Running a CI server is not the same as doing CI properly,” said Andrew Phillips, VP of product management at XebiaLabs. “To a lot of people, the fact they have Jenkins running magically means that they check the CI box. If you ask whether they are continuously integrating all of their features to figure out where their master is in an integrated state and actually releasable, you hear, ‘Of course we don’t do that. We have a master but that’s way behind.’ ”
A financial service customer of Electric Cloud has 38 applications subject to Sarbanes-Oxley requirements that are still built manually. The company estimated that it will take two years to roll out CI across all of them.
“There is still a greenfield business opportunity out there,” said Anders Wallgren, CTO of Electric Cloud. “It’s just going to take a while before everybody completes the transformation.”
Achieving CI at scale is tough
Scaling CI can be a challenge because what works well for an individual or team may not work as well or at all for multiple groups or across the enterprise. Organizations are speeding up the build process and requiring developers to check in early and often. As a result, the number of check-ins may have grown exponentially. Atlassian is doing 300,000 builds a month, for example.
“We didn’t magically flip a switch and say it’s easy now. It took years to get there,” said Atlassian’s Wittman. “It’s apparent that some teams aren’t even doing best practices yet. Once you get over that hurdle and see the value, you see things like NASA using Bamboo to deliver mission plans to the Mars Rover.”
As teams move from automating builds to CI, they may overlook factors that can negatively impact productivity later.
“What happens is something I call single-box CI,” said Sumit Mehrotra, director of product management at Skytap, which provides on-demand test environments. “When a check-in happens, I spin up one box that has all the components of my application running at as small a scale as possible, and then I deploy my code and run a gamut of tests on that box. That’s relatively easy.
“But if you talk about integration, it’s about bringing real components together and then testing your code. In that sense, they haven’t gotten to a point where they can actually run full-scale CI.”
Although CI may work a particular way for an individual team, the dynamics change at the enterprise level.
“It’s massively more complicated to do CI at an enterprise level because you need lots of little components talking to each other to do your testing,” said ThoughtWorks’ Mason.
Of course, those dependencies are not the only internal component dependencies given the common need to use third-party components.
“The package dependencies have become a nightmare for teams,” said Atlassian’s Wittman. “I watch our teams go through this. They’re using all of these third-party libraries. What version are they using? Is it the right version? Oh, they can’t use the latest version because of some issue, so they have to use the older version. All of those dependencies throw people off.”
Tracy Ragan, chief operations officer of OpenMake, said scripting can also be problem, particularly at scale.
“[Software teams] build these scripted processes around CI, then they do deployment and they write scripts to do that,” she said. “They may even have some scripts automatically update JIRA, for example, and for one team that works really well. When you have two or 10 teams, or 50 to 100 in an enterprise, you have a centralized team that has to manage the scripts for multiple people, so you need to move from a script-driven process to a model-driven process.”
One of OpenMake’s customers, a credit card company, adopted a model-driven process after developers experienced centralized script management first-hand. One member of each development team was required to work the swing shift, when the scripts broke most often. All of them were required to release someone else’s code instead of their own so they could experience what it was like to manage scripts when one is blind to what is in them.
“They went from thousands of scripts to 10,” said Ragan. “And they reduced their release team from about 40 to half of that.”
Embracing CD
CI can only take teams or an enterprise so far. While accelerating builds and test automation is widespread, not everyone has embraced the notion that software should be ready for delivery at the end of an iteration.
“Business requirements are changing very quickly, so you need to be able to react faster,” said Atlassian’s Wittman. “Instead of being in reactive mode, you need to get the code deployed to customers faster. That’s a game-changer for software teams. It requires a culture and discipline that many software teams are still in the early phases of adopting.”
The added speed benefits of CD can be dramatic. For example, FamilySearch, an Electric Cloud customer, had a 90-day delay between the time engineering completed a change and the time the change appeared on its website. After embracing CD, the company could deliver software in 10 minutes.
“Continuous Delivery is having a pipeline of phases through which your software travels on its way to production. CI is a first step in that process,” said ThoughtWorks’ Mason. “You layer on all of the other kinds of automated tests—integration, performance, and acceptance—going down the pipeline into production, and then it’s a business decision whether to put it into production or not. Continuous Delivery is an IT capability that allows a business to move at the pace they need to.”
ThoughtWorks recently introduced Go, a CD tool that automates the build-test-release cycle.
“Some teams doing CI are still doing traditional QA,” said Mason. “You can’t get away with long, manual testing cycles if you’re trying to do Continuous Delivery, and you can’t get away with production deployments that require everybody to work over the weekend.”
CloudBees is delivering Jenkins as a service. The company recently teamed up with Pivotal to offer collaborative, enterprise-grade CI and CD solutions. In addition to providing Jenkins support, its Jenkins Enterprise service includes plug-ins for security, high availability, Jenkins management and administration, and resource utilization.
“The ability to operate at scale is a key problem facing large organizations moving to Continuous Delivery,” said Steven Harris, senior vice president of products at CloudBees. “On one hand it means just handling the resources to get the job done, [but] even more importantly it means moving work and artifacts across a complex organization, and how to use Jenkins to help bridge dev and ops teams.”
Whether an organization embraces CD or continuous deployment depends on the business model and customer requirements. While the goal of CD is to get software ready for delivery, continuous deployment means committing and deploying.
“Continuous deployment is the much scarier version where every change that a developer commits ends up in production after it’s passed all the automated tests,” said ThoughtWorks’ Mason. “Etsy and Netflix want all their code in production all the time, and they have incredibly robust test suites that ensure their systems are working.”
The growing popularity of CD and continuous deployment is aided by the DevOps movement, which requires development and operations teams to work closer together than ever before.
“The definition of a software team now includes the Ops guys, whereas four to five years ago it didn’t,” said Atlassian’s Wittman. “The Ops guys represent what it takes to ship software to customers.”
While continuous deployment is integrated into the DNA of companies like Facebook, CD is more practical for the bulk of organizations.
“You want to get to the point, regardless of where the software is going in the end state, that you’re always in a release-ready state, that you’re always ready to ship your software,” said Electric Cloud’s Wallgren. “That allows you to decide when to ship and what to ship based on business needs rather than it’s broken and we can’t ship it.”
Not everyone agrees with the business value of potentially shippable software, however.
“Agile, Continuous Integration and writing potentially shippable code sounds great. The value of potentially shippable product is arguably around zero because if your customers aren’t using it, how can they benefit from it?” said XebiaLabs’ Phillips. “Some mobile app developers throttle their release process to meet user requirements. Don’t get swept away by the hype. Put it in the context of what your customers actually want and what makes the most sense for your business.”
Avoiding infrastructure gridlock
Having enough resources is a huge issue for companies embracing CI and CD.
“Many organizations treat the cloud as a way to get boxes faster, which only gives them 5-10% of the value. You need a high level of automation, which isn’t just build automation but infrastructure,” said ThoughtWorks’ Mason. “We have an infrastructure-as-code technique where you can actually use version control to manage all of your infrastructure configuration, including firewall configuration and load balancer stuff. You can put all of that into source control as well as your application code and manage it in a consistent, repeatable way.”
The problem with a shared build infrastructure is that mission-critical projects need dedicated resources, but sometimes they get stuck in a queue.
“We’re trying to give those specific projects dedicated resources they can use but still have a shared build infrastructure,” said Atlassian’s Wittman.
The sheer number of builds and tests being executed in parallel taxes the centralized test environments some organizations have created. If the test environments are saturated, builds or tests can’t be executed simultaneously. And even if the resources are available, some organizations aren’t able to provision them in a consistent manner because they lack the tooling to do it.
“It’s hard to share golden testing environments especially in the context of distributed teams throughout the world,” said Skytap’s Mehrotra. “How can you ensure that all of your guys are working on the same exact setup so the work they’re doing actually matches? How can you ensure you can test early enough in the development cycle and run test in a representative environment that looks more like a complex production environment?”
To solve the problem, Skytap Cloud provides self-service access to consistent environments on demand. It is integrated with Jenkins so users can define team workflows and avoid common on-premises bottlenecks.
ThoughtWorks has also approached infrastructural challenges, albeit differently. It built a system for The Gap, which uses Google Chat integration. The solution allows users to send a message to a server from within Google chat instructing it to deploy a build. The server then spins up a virtual environment with that particular build deployed and reports when the build was done. It also includes a URL that can be used for a stakeholder showcase or QA.
Managing cultural shifts and tools
Continuous ways of working require adjustments to processes, workflows and work styles that can make some individuals uncomfortable, whether manual processes are being replaced by automated processes, or because territorial boundaries are being superseded by cross-functional collaboration.
“Operational change is hard because there are organizational fiefdoms, and you get people like the QA manager who likes to be in charge of 50 people,” said ThoughtWorks’ Mason. “The problem with organizational silos is that they impede agile practices, so you need to get the entire organization to challenge some of those structures and how they work.”
Rather than implementing a new feature and throwing it over the wall, software teams have to consider how the product is going to evolve over time, who should own the product, and how to wring the greatest amount of value from investments.
“I will certainly sell you CI in a box, but I promise you you’ll fail if you look at it as a product you can just get and be done with,” said Electric Cloud’s Wallgren. “You have to have sponsors for the transformation, air cover from management, and have people lined up to do it because if you’re going from manual builds to CI, Continuous Delivery or continuous deployment, it is going to change the way you work on a day-to-day basis.”
In large, established companies, different teams often establish different processes that are hard to reconcile when there is an enterprise mandate to adopt something like agile development, CI or CD. Meanwhile, there may have been mergers and acquisitions involving people who represent different cultures and experiences. In addition, the tool choices among teams often vary, and the nature of the projects the teams work on also varies. As a result, each team’s level of agility can vary significantly depending on those and other factors.
“Getting buy-in from everyone is difficult. If you try to force this sort of one-size-fits-all process and toolset, it’s doomed to failure,” said Wallgren. “You’ll get resistance because the [existing] tool works and it’s getting the job done. The tools need to be integrated into a process that ensures the tools are being kicked off automatically so that you don’t have manual steps in your processes.”
XebiaLabs’ Phillips agreed. “Tool chain choices are important. A key part of the decision in a large organization is how well your choices mesh with your existing investments and processes,” he said.
“Using your corporate standard for source-code control and approval processes while doing CD isn’t even an option for companies under strict compliance regimes. This is another reason why Jenkins adoption is growing so quickly. People see it as a tool that enables CD, which can be extended by them to work with their existing investments.”
As always, automation makes people nervous because it pares down or obliterates certain job descriptions. However, as software teams and the codebase grow, automation becomes critical.
“If you’re a large organization and you have diverse tooling, you need a process that stitches them together in a seamless fashion,” said Skytap’s Mehrotra. “At the same time, you want a workflow that allows you to get ready for release as fast as possible and with the highest quality possible.”
Concurrently, the right information needs to be delivered to the right people at right time, whether it’s the status of a build, the resources a particular engineering team is using, the time it will take to procure the resources engineering needs, the status of the release, or the quality of the software.
“A lot of those questions fall into the category of visibility and control,” said Mehrotra. “We have a great solution, but there is still a lot that can be done, especially in the context of large, diverse development organizations.”
Towards 2015
Build and release cycles will become more integrated and automated in the near future as more teams extend out from CI to CD. OpenMake recently introduced Release Engineer, a release-management tool that will be integrated with its Meister build-management tool in 2015. The integration will allow Meister to automatically update the packaging component of Release Engineer. Other solution providers plan to enhance their products, placing more emphasis on CD capabilities.
“Customers want to do less work and be able to onboard more application teams faster and with more control,” said OpenMake’s Ragan. “The way to do that is to automate as much as you can.”
Tools can only take an organization so far, however. Transformation requires a change in mindset.
“You need to have buy-in at the highest levels of the organization,” said Mehrotra. “Already the ability to deliver software faster has demonstrated the value of agile software development to CIOs and VPs. If a whole organization is going to reap the benefits [of CI and CD] not only does an individual team need to buy in, other teams have to buy in, especially the people associated with or managing one or more pipelines in the organization.”
Adopting an agile mindset can help by embracing continuous improvement that enables fast change on a small scale.
“Agile was about getting developers and QA to talk to each other,” said Wallgren. “Continuous Delivery and DevOps extend the conversation to Ops people and release management, so we’re really starting to think about this as an end-to-end process from check-in to release. More CTOs and product owners are thinking of Continuous Delivery, not just CI.”
