Companies want to deliver high quality software fast, but how fast is fast enough? A recently released report by CircleCI found three important development metrics to predict an organization’s velocity and maturity. The metrics are: mainline branch stability, deploy time and deploy frequency.

“Most teams have no idea what speed actually means,” said Jim Rose, CEO of CircleCI. “They are deploying and they are doing it at a rate that feels good to them, but they don’t necessarily know where that stacks up compared to the rest of the market.”

The study looked at a sample of organizations built on the CircleCI’s cloud platform from June 1 to July 31, 2017.

Mainline branch stability refers to an application’s “source of truth.” According to Rose, the question you need to ask teams here is “If you had to release the most recent version of your code right now, right this second, could you? Is your mainline green, and can you deploy whenever you need to?”

According to CircleCI’s report, 80 percent of organizations keep their mainline deployable 90 percent of the time. Organizations in the 95th percentile can deploy 99.9 percent of the time. However, the lower 5th percentile are only stable 53 percent of the time compared to the median, which is 98.5 percent of the time.  

The second question you need to ask teams is “How long does it take between the time you make a commit until it gets into the data center?” According to Rose, the less time it takes to deploy, the less expensive it will be to make changes to your product. In CircleCI’s study, the company found 80.2 percent of organizations deploy in under 1 minutes. The 95th percentile are able to deploy in about 2.7 minutes with the 5th percentile taking 30 minutes. The median found here was 7.6 minutes.

After a team stabilizes their mainline and lowers their deployment time, the next key question is “How often do you deploy and how quickly are you pushing changes?” In the study, the company found 75 percent of organizations deploy their most active projects less than 13 times a week. The top 95th percentile deploy 32 times a week, which is 5 times the median and 24 times more than the 5th percentile.

“We see that deployment frequency is a key indicator of high performing organizations,” Nicole Forsgren, author of Accelerate and CEO/Chief Scientist of DevOps Research and Assessment (DORA), said in the report. “High performers can deploy on demand, while their low-performing peers can only deploy their code once per month or longer. This difference in deployment speed makes all the difference in delivering value, delighting your customers, and keeping up with compliance and regulatory changes.”

Once a team finds out where they fall in comparison to their competitors, they can use those metrics to start to improve, according to Rose.

In the mainline stability case, a way to do that is to make sure you have a good process in place before you ever merge anything into your mainline. Historically, teams are use to merging their code into the mainline and just figuring it out as they go along, Rose explained.

“You need to make sure you have a great code review process, pull review process, and a robust set of tests that are automated and constantly run every time someone is making a commit so that you know well before you merge into the mainline that the merge is going to work,” Rose said.

Other best practices for mainline stability include efficient recovery from failure, communicating the cost of failure, and communicating potential conflicts.

To improve deploy time, Rose explained that any human based action within the system should never have to wait for a machine and organizations should figure out how to use as many machines as possible to keep time down and keep developers productive and happy.

“One byproduct of having a robust test suite is a reduction of time spent performing manual QA. Organizations can only do this when processes are highly optimized: few bugs, efficient recovery from failure, and constant monitoring,” the report stated.

Lastly, for deploy frequency teams should practice small, short-lived pull requests and use feature flags.

“Frequency of deploy is ultimately the health check for the rest of your system,” Rose said. “You want to get a change into the hands of your customers as fast and as much as humanly possible, and you want to know that it is going to deploy safely.”