Continuous integration has become a staple in many organizations because it allows software teams to improve product quality, save time, lower the number of failed deployments and reduce rework costs. Despite the benefits of faster code iteration, not all software teams have embraced the practice.
According to Tom Grant, a senior analyst at Forrester Research, approximately two-thirds of software teams claim to have adopted continuous integration, while the remaining one-third has not.
“It’s hard to do continuous integration,” he said. “Although it provides immediate benefits, it requires a [serious] commitment.”
One reason so many organizations have adopted continuous integration is to avoid unnecessary rework.
“If you’re integrating a week or two’s worth of changes and trying to retrofit or retrofix the backlog, [you’re wasting time],” said Mike Rozlog, senior director of Delphi Solutions at Embarcadero Technologies.
“If you’re constantly updating, you have a huge advantage from product quality and rework standpoints. The longer you hold onto code, the more rework you’re going to have to do. You want to know that your code is okay and won’t break someone else’s.”
All organizations can benefit from continuous integration whether they are agile, non-agile or some sort of hybrid, because it allows developers to isolate and fix errors faster and cleaner, with less overhead.
“Continuous integration is the ultimate safety net,” said Chris Clarke, VP of product management at CollabNet. “Because teams are getting leaner and meaner, there are [fewer] QA guys, so developers need more feedback. In the past, you could spend months putting the guts back into your code. Now, in minutes or hours, you know if your changes will negatively impact the software you’re integrating with, and you can be more confident about your code.”
For agile teams, continuous integration can help increase code output, improve product quality and reinforce disciplined software practices.
“Continuous integration is a core piece of agile,” said Thomas Murphy, a research director at Gartner. “There are rules for check-in and what needs to happen in testing before check-in.”
If organizations want to advance to “true” agile (as few organizations are purely agile), continuous integration is a must because it is important to know as soon as possible whether code is good or bad, said Rozlog.
In non-agile organizations, continuous integration forces teams to make a greater investment in builds and testing. According to Grant, team dynamics tend to improve as a result of continuous integration, especially in organizations with waterfall practices, because the entire team is committed to changing what happens early in the software development life cycle.
Usman Muzaffar, VP of product management at Electric Cloud, considers continuous integration as useful for waterfall practices as it is for agile practices because of the visibility it provides.
“There are good reasons why continuous cycles are longer [in organizations with waterfall practices],” he said. “Even if you’re moving from months to weeks or longer, you can still get the benefit. Until you can see your code playing nicely with others, you’re flying blind.”
For hybrid organizations, the most common question is which of the many application life-cycle management disciplines should be adopted. If they do nothing else, said Grant, they should adopt continuous integration, because it is a discipline that tests mettle by requiring everyone on the team to succeed often, if not every day.
Regardless of where an organization is on the agile continuum, one of the major benefits of continuous integration is feedback. Without continuous integration, developers sense they’re making progress but often fear the rework that will be required days, weeks, or months later when they finally discover how misaligned their changes are, said Muzaffar.
Central to the notion of “continuous” is automation because it saves time, speeds processes and lowers costs. Maciej Zawadzki, president and founder of Urbancode, said that although it is possible to automate using a Perl script, to truly realize the benefits of automation, software teams need two additional things: visibility, so they can identify problems and fix them; and a self-service aspect, so they can run scripts themselves even if they are unable to see someone else’s code.
“It’s not a question of failure because failures are going to happen,” he said. “The more transparent failure is, the easier it is to diagnose what failed.”
Continuous integration is a discipline first and foremost, which means software teams need to commit to a common set of practices, such as smaller and more rapid code iterations, and regular, frequent builds.
“The team has to invest in doing something that’s different up front,” said Grant. “There’s always someone that says his or her work can’t be broken down for some reason, like there’re too many dependencies. If you want the team to move in the same direction, you need to make a common investment in the outcome.”
Zawadzki agrees, adding that some fear the feedback they’re going to get when builds break, either because it will imply the codebase is unstable, or that the developer really isn’t as good as everyone assumed he or she was.
Some teams are concerned that if everyone is focused too heavily on a single thing, like build scripts, then the team’s overall performance will suffer. However, an even larger barrier is a cultural one.
“Continuous integration requires a change in developer behavior,” said Gartner’s Murphy. “In some cases, the problem is inertia because people are doing what they’re comfortable with.”
There are also concerns about initial capital and operating expenses. On one hand, there is the cost of tools and hardware, and on the other hand, there is the cost of altering developer behavior.
“Setup and maintenance is a huge issue,” said Embarcadero’s Rozlog. “You have to consider structure, architecture and personal biases, and then decide whether the overhead is worth the supposed benefit.”
Zawadzki said some of his customers have taken continuous integration to an extreme, because they’re operating at such a fast pace, it is worth the cost of dedicating 100 machines to run builds and tests if it alleviates the cost of a bad, buggy commit.
In complex environments, among teams managing lots of branches, or in organizations that have moved beyond nightly builds to truly continuous builds, continuous integration can become a “box hog,” according to CollabNet’s Clarke and Mike McGarr, a senior consultant at Excella Consulting. Solutions to the problem include notifying the group affected in advance of a build, dedicating machines to builds, or taking advantage of the cloud.
Best practices for adopting nightly builds
Not all organizations that have adopted continuous integration are doing nightly builds, and even those that are doing nightly builds may also be doing more frequent builds to keep the codebase as up to date as possible. Setting nightly builds as a goal is an important step because it requires discipline and helps reduce rework.
“A lot of us started out with nightly builds because it’s easy to get started and it forces you to get into good work rhythms,” said Clarke. “When your organization matures, you can move to a commit-by-commit basis.”
One Urbancode customer moved from nightly builds to four builds per day so teams in different regions could see the results. Other organizations have taken advantage of nightly cycles to run automated tests, do static analysis, and execute security scans in addition to builds.
Nightly builds can help ensure any misalignments are less severe than they would have been had the builds been run less frequently. Moving to nightly builds can also help teams become more disciplined.
“Nightly builds are fairly well adopted. If your organization is agile, you’re striving toward more frequent iterations, so you want to get your organization into a daily rhythm,” said Gartner’s Murphy. “You may also decide to have a peer review once a day. Continuous integration means varying things that go on with builds. If you can get into a regular rhythm, you can be more consistent.”
Nightly builds and builds that occur more or less frequently are not necessarily mutually exclusive. Some organizations are initiating builds every time code is checked in, and they are also doing nightly builds. Nightly builds are also being combined with less-frequent builds that may involve more in-depth testing.
How often builds occur may depend on the length of the build cycle; if the build takes more than 24 hours, then nightly builds are impractical as structured.
Common best practice recommendations:
Set team expectations. Because nightly builds require constant consumption of and contribution to code, developers need to get used to smaller, faster, more iterative cycles. Early on, you will likely notice code-quality improvements. Later, you will likely discover that product quality and processes are becoming more predictable and that you can use metrics to further optimize them.
Get into a rhythm. “Continuous” can mean literally continuous, nightly builds, weekly builds, other periodic builds, or a combination of builds. Regular, disciplined practices help ensure product and process consistency. Without discipline, expect continuous breakage. Also, make sure your test cases are prioritized so the build can run within the confines of its cycle frequency.
Run builds as frequently as possible. That way, developers can get access to the most recent version of code and reduce the amount of rework. You can help speed things up by automating the build process.
Have an infrastructure in place. Continuous integration is best achieved using automated tools, build servers (vs. developers’ desktops), and other infrastructural elements that can help facilitate its success.
Try preflight builds. One common gripe about continuous integration is that it is post facto. Some tools allow you to test the effect of changes prior before you officially commit. (This is particularly useful for branched code.)
