Measuring developer productivity isn’t as simple as it sounds

When developers lie awake at night, they’re likely not thinking that they didn’t turn around enough tickets that day, or write a certain number of lines of code. Their fear is that they’ve broken something, and that they’ll be in trouble.

In the meantime, C-level managers are primarily concerned with innovation, creating new products and enhancing old ones. So there is a natural divide when it comes to assessing how productive developers are.

Eric Minick, vice president and head of product at CodeLogic, said, “I think what a lot of our developers would celebrate most is, if someone said, ‘Today, I took 300 lines of code that were a mess, and I consolidated it down to 40 lines of code that are clean. And so my net code for the day was minus 260 lines. And that would be celebrated wildly. And so lines of code is about as toxic a measure as you could come up with, as it encourages bad behavior.”

RELATED CONTENT: How CodeLogic helps make developers more productive

“You know, a development team or an IT shop, looking at that developer who took that 300 lines of mess and turned it into 40 lines of elegant clean code, as being productive,” he continued. “Somebody in a business suite, however, might say, ‘You are not advancing our product, you added no new features, nothing happened. How are you being productive?’ “

Gartner analyst Thomas Murphy explained, “We advise clients that they should not be focused on individual productivity metrics – software is about teams – thus we look at team productivity and things are measured more in agile terms of Story Velocity, but that is useful more to understand a backlog and how long it will take.”

It’s this misalignment between the business and IT that continues to exist — despite the ideals of Agile development and DevOps that should bring the sides closer — that makes defining developer productivity difficult. It is Minick’s opinion that from a measurement point of view, most development teams have yet to make business outcomes their goals. Organizations might be using OKRs or KPIs to say that in the next six months, we’re going to improve conversion rates by 5%. But developers are saying, ‘We’re still closing 100 tickets a week, we are good at our jobs.’ Minick said, “I don’t think most organizations have really tightened up the alignment to bring the business metric into the definition of success for the application team. But we’re starting to see the beginnings of that.”

Gartner’s Murphy said, “From a metrics perspective you should be shifting away from metrics that are ‘output’ driven and to metrics that are ‘outcome’ driven.  Thus are we delivering the business outcomes – which means it isn’t just an engineering thing.”  

One thing that organizations have started to embrace in an attempt to make developers more productive is the notion of the developer experience, with the belief that giving developers the best possible employment experience increases their productivity. That experience can range from things like the chair they sit in, the size of the monitors they use while working, the software tools they are given to do their jobs, and the hours they put in.

“There’s nothing more frustrating than having to close down a bunch of apps just to start running your tests and get your own software to run, and being constrained by a cheap laptop, or a monitor that’s too small or anything like that,” Minick said. “You want loyalty from your developers. Give them a powerful box and a big screen, like step one. Step two … big investment in better chairs, standing desks, all of these things that set up the developer to be comfortable, alert, helping and able to concentrate for a long time on their code and be effective.”

After that, he said, make sure they’ve got the right tools at their disposal. Make sure they’ve got a good development environment, that they’ve got the other software packages they need.

Changing roles for developers

One of the difficulties in assessing developer productivity is the fact that their job is much broader than it was in the days when developers primarily wrote and maintained code. Now, they’re more involved in testing, more involved in security and in compliance and governance.

Minick said measuring things like features delivered by the development team is better than measuring the amount of code generated. And, organizations that take productivity seriously will put in place measurements for ‘good behavior,’ such as how code coverage is changing, and is technical debt increasing or being reduced. Or, he noted, crediting developers for taking something highly complex and streamlining it to something simpler. While there was no feature added by that work, the technical debt score should go down, and that would be the productive activity.

A productive development team, according to Minick, will deliver features, mitigate risk, and fix bugs. “You want to make sure you’re balancing your investment in a development team pretty well across those things,” Minick said. “If you’re delivering no features, you’re probably failing. At the same time, if you’re delivering only features and accumulating a tremendous amount of technical debt and risk, you’re setting yourself up for failure in the future. And it’s really a business decision how to weigh that investment, but that should be done consciously, and too often, it’s not.”

How to make development teams more productive

One of the ways organizations can increase feature flow is through value stream management, with which they can identify the impediments that slow productivity and work to remove them. “So metrically, this is where the DORA metrics come in, or the Flow Framework, and this is what comes into a ‘Value Stream Management’ system,” Murphy said, “but with this you are also looking at what are the bottlenecks. ‘Ahh, it takes two hours to provision a test system, that limits how quickly we can build and test software — how can we make that faster.'”

This is especially true as it pertains to the various tools developers need to be productive. There are communication tools such as Teams and Slack, CI tools, IDEs, test tools, code repositories and security tools. Some of this tool sprawl is necessary, Murphy said. “I have to have an IDE, a compiler and such,” he said. “The hope would be that I don’t have Teams and Slack and email for communications.”

Many organizations have more than one tool for continuous integration, which could be because teams have the freedom to select the tool they feel is best to do their job. Murphy noted that Gartner has seen a shift toward choosing standard solutions and tools that take a more integrated approach. “More clients are buying Jira/Confluence/Bitbucket/Bamboo… than did in the past, where it may have been Jira/Confluence/Git-something/Jenkins/Artifactory,” he said, adding that this will be a slow evolution, because of sunk investments and personal preference, but that organizations want to control their spend, be more efficient and have the ability to move resources between teams.

This is another aspect where the business and IT seem to be at odds, though organizations are recognizing the value of having business knowledge in and with the technology. The way many of them are meeting this challenge is through retraining. CodeLogic’s Minick said that’s the best way to go, as opposed to hiring more developers, “because you have the knowledge of business in those people, and you want to keep that in house.” This approach can keep the deep business understanding of how the business works as close to the developers as possible.

Gartner’s Murphy said organizations do a mixed job of providing time and resources to support learning and upskilling, which is leading to growth in informal training — communities of practice, dojos, use of StackOverflow, and more use of pairing and mentoring. “Entities are having to rethink strategies and tools to support how new people are onboarded, how to get effective knowledge transfer and how to evolve employees,” he said. “But this also means you have to be prepared to train them in the new hot technology, pay them more, or (risk losing them).”

A matter of trust

It’s critical that IT and the business have trust, so that when the business asks IT why something will take six months to complete, the answer is framed in such a way that the business can decide if it wants to proceed down that path or not. According to Minick, you’re starting to see more product management people put into that intersection.

Business talks about how quickly features and bug fixes can be released. But developers talk about velocity. Velocity, as any student of physics would recall, is different than speed. “Velocity is a vector, velocity has direction,” Minick said. “And what we really need is speed in the right direction. That’s why people are pulling in the SME [subject matter expert] knowledge to business, to better make sure that when we write a lot of code … it’s what the business wants — and more specifically, what they need. And that often requires someone who has a good understanding of the business to translate from what the business says, which may not be precise enough or clear enough. There may be very ambiguous language to actual features that development can build that will actually push the application and therefore the business in the right direction.”

The crushing demand for modern apps

The demand by the business for new applications and modern experiences far outstrips the availability of developers to do the work — a labor shortage that has plagued IT in the United States for nearly two decades. Combine that with the growing complexity of modern applications and the near-constant introduction of new platforms, technologies and modes of interaction, and — Gartner’s Murphy said — overloaded engineers “have a  feeling of, ‘Here, let me toss a few more bricks on the hod.’ 

This, he said, is part of the reason that there is such growth in low-code solutions, to ease the burden on “pro-code” developers. The solutions, Murphy said, support “that fusion team where developers create new services and components and citizen developers compose those into business applications.”

Keep it as simple as you can

In the face of all this, Murphy suggests organizations do what they can to simplify things. Among the steps they can take are reducing the number of tools where possible, and being consistent in business direction/objectives so you have a unified set of activities that everyone is actioning toward.

Going forward, organizations are looking to AI-assisted coding to help with productivity, but Murphy noted that will “take a while to perfect.” In the meantime, he said, “toolchains will be a bit messy, and our ‘legacy challenge’ of unforeseen needs of the future will always dog us.”

A guide to developer productivity tools

Developer productivity encompasses the use of many different kinds of tools — a comfortable chair and two big monitors, for example. On the software side, developers need tools for writing code, integrating code changes, testing, putting security around their work, and more. What follows is a sampling of tools in each productivity category.

Code repositories

• Bitbucket
• ClearCase
• Codebase
• Eclipse
• Git
• GitHub
• GitLab
• IntelliJ IDEA
• Java.net
• Netbeans
• Perforce
• SourceForge
• Subversion
• Visual Studio 

DevOps platforms

• Atlassian Bamboo
• AWS Code Pipeline
• Azure Pipelines
• Bitbucket Pipelines
• Broadcom
• CircleCI
• CloudBees
• Codefresh
• Digital.ai
• Harness.io
• Jenkins
• LaunchDarkly
• Optimizely
• Split Software
• TeamCity

Collaboration tools

• CodeLogic
• Codenvy
• Codestream
• Engine Yard
• JetBrains
• SmartBear

Communication tools

• Google Mail
• Google Meet 
• Microsoft Teams
• Slack
• Trello
• Zoom

Testing tools

• Applitools
• Applause
• Eggplant
• HCL Software
• Mabl
• Parasoft
• Progress Software
• Sauce Labs
• Testim
• Tricentis

Security tools

• Aqua Security
• Bridgeview
• Checkmarx
• Contrast Security
• Sonatype