There are three ways to create a DevOps culture: by default, by design or iteratively. The three are not necessarily mutually exclusive because DevOps tends to be a learning experience that takes considerable time.
For example, talent marketplace Topcoder has had a formal DevOps practice for the past three or four years and started a DevOps-like practice in 2007. As part of its transformation, the company moved from an on-premises data center to the cloud and rebuilt its applications as cloud-native applications. More recently, it changed its tooling and updated its DevOps practice to include security and compliance.
DevOps is ultimately a combination of a culture, processes and tools, although not everyone sees it that way. In fact, there are a lot of misconceptions about what DevOps is.
“DevOps is a set of practices that give organizations and teams unified, cross-functional representation, shared accountability from development, operations, the business and everybody in between,” said Justin Rodenbostel, VP of delivery at digital transformation agency SPR. “People work together throughout the development process to create higher quality software solutions in shorter time frames. We like to think of it as an extension of Agile.”
RELATED CONTENT:
Dynatrace replaces DevOps with NoOps
CI/CD pipelines are expanding
Still testing like it’s 1999?
Charles Betz, principal analyst at Forrester, said that “building a DevOps culture” is a misnomer, however.
“I think a subtle misconception is that you can somehow start with culture. This may be contrary to what you’re hearing from others because culture is important. But I believe as do a lot of other folks, that culture is a lagging indicator,” said Betz. “You do change culture by changing practices. You do change culture even by changing tools.”
Changing culture takes time because it involves changing mindsets and ways of working. While one can start using a new tool today, it may be several years before an organization believes it has achieved any kind of maturity. In fact, many of the companies interviewed for this article series have been practicing DevOps since 2015 or longer and consider themselves to be either at the beginning stage or the “awkward teenager” stage.
“You don’t build the culture, it’s more emergent, more organic,” said Forrester’s Betz. “It gradually manifests as the sum total of all the interactions people are having and how those interactions are being transformed because they now have new operational capabilities that they didn’t have before and they have permission to use those capabilities.”
Not everyone agrees with that outlook, however. In fact, some practitioners and consultants advise starting with a cultural end state in mind so it’s clearer how processes, practices, and tool sets need to evolve. That said, even with a planned transformation, pivots tend to occur along the way as organizational priorities, competitive factors, customer expectations, internal dynamics and technology all change.
“Now we’re working more closely with the business. We’re communicating better, we’re adapting to change better, and we’re delivering what the business is expecting in a timelier fashion,” said SPR’s Rodenbostel.
How roles are evolving
Forrester recently published a report entitled, “The Future of Technology Operations.” In it, Betz observes the flatter organizational structure of modern technology operations versus the traditional hierarchical structure. The flatter or matrix structure of the organizations impacts the roles of individuals and team dynamics.
“The idea that somebody can specialize is falling by the wayside,” said Betz. “You get to the T-shaped people who each have a specialty and a certain element of breath. You have more line of sight, a lot more interest, a lot more commitment to the mission outcome of a team and you have somewhat less commitment to your professional identity.”
That can be an uncomfortable transition for people whose self-worth is tied to a particular role.
“[Traditionally,] goals and responsibilities have been based on a person. They have to be based on the success of the team,” said Rodenbostel. “When roles are being invented to align with DevOps, there is some discomfort because it’s more of an environment of shared accountability. Whereas when you operate in silos, it’s much easier to be successful as an individual.”
DevOps and CI/CD pipelines provide visibility into what’s happening throughout the development process, which has three effects. The first is providing individuals with insight into their own progress. The second is facilitating a sense of shared ownership and responsibility. The third is making people accountable for their work.
At talent marketplace Topcoder, deployment engineers became DevOps engineers. And at messaging services company Gupshup, developers have begun taking responsibility for writing deployment code. QA has moved from performing testing to maintaining automated test suites.
Dominic Holt, CTO of fractional CTO company Valerian Tech, said he first goes to the deployment files and automation scripts before writing a single line of code so he knows the code will work in whatever production environment he wants.
Teams are evolving
Like Agile, DevOps is associated with cross-functional teams.
“One of the foundational pillars of DevOps is systems thinking,” said CTO consultant Emad Georgy. “[Instead of asking] who’s fault is it, systems thinking looks at the whole system. If something went wrong with the pipeline, no matter who owns it, let’s look at the system as a whole and understand what the actual root cause is and solve it. A lot of people don’t do that. A lot of people are not ready for that kind of thinking.”
Teams are self-organizing at business and technology services firm Orion Business Innovation.
“It’s about empowerment. They have motive, they have purpose,” said Pradeep Menon, EVP at Orion.
Trust is a hallmark of an effective DevOps team. Specifically, members of the team trust each other. There is also a technical aspect of trust. Organizations are placing trust in that which is automated, whether it’s policies, deployments or a database’s ability to self-heal.
“I think one way to encourage behavioral change is by doing regular measurement and responding to the results of that measurement,” said SPR’s Rodenbostel. “Give the team the opportunity to improve when things aren’t going well and celebrate when things are going right.”
Another wise move is aligning financial incentives with the desired behavior, which not all organizations do. Part of the problem is that business leaders and HR don’t always understand the subtleties of IT. The disconnect can work against IT cultural transformation.
At Topcoder, everyone can see the statistics, which makes it easier to see how code changes impact revenue, for example.
Governance is evolving
DevOps involves a lot of automation throughout the software development life cycle. With infrastructure as code and even compliance as code, governance can be integrated into the pipeline in a manner that does not add an unwieldy amount of overhead.
“If you have five product lines, all with their own product teams and release cycles, and they’re all released in their own way, [you may benefit from] establishing one standard DevOps pipeline where everybody builds, integrates, tests, deploys and monitors the same way because everyone’s using the same pipeline,” said Georgy. “The security guys are elated because they no longer have to look at five products that all have their own custom builds; they can secure one standard pipeline.”
Forrester’s Betz said governance needs to switch from how to what.
“You need to govern the promises your team is making,” said Betz. “It takes a certain level of trust and holding teams accountable for what they said they were going to do. It’s basically elevating the concept of internal SLAs but then not really meddling beyond that, unless you need to do forensics on why something really went badly.”
Of course, that approach is at odds with auditing, which demands an explanation of exactly what happened, why it happened, when it happened, and who was involved.
“The trouble is that work is getting done in a much more collaborative, less deterministic, fuzzier way. So, when an auditor comes in and says show me your standard flow chart for how this work is supposed to happen, there is no flow chart,” said Betz.
How to get started
If you haven’t started your DevOps journey yet, it’s wiser to start with a pilot project and learn from that instead of attempting a massive transformation effort from the start.
“If there are multiple products in the portfolio, [identify] which could immediately benefit from a DevOps transformation,” said Georgy. “Then go after quick wins.”
One thing to keep in mind is even if a pilot went well, it may not scale as-is perfectly to another product or team. The approach may have to be adapted to suit the goals of the team, the people on the team, and the business’s expectation of the team.
“It was relatively easy to implement DevOps for a specific product where it was the only way to achieve our desired outcome,” said Nirmesh Mehta, CTO at Gupshup. “It was much harder to expand it to other areas where there was no burning requirement and the possibility of a lot of disruption.”
One question is whether it’s better to have dynamic teams or static ones.
“One of the most important things that the modern manager is starting to realize is that a high-performing, cross-functional team is a very precious resource, and you don’t just throw people on or off those teams,” said Forrester’s Betz. “You don’t just take Mary off the team and replace her with Myra because Myra knows Python.”
If Mary has been a contributing member of a team and understands the business problems the team is attempting to solve, then It’s much better to teach Mary Python, assuming she can wrap her head around it, Betz said.
SPR’s Rodenbostel recommends considering the product that’s being worked on, the tools already in use, people and their skills as well as organizational constraints.
“If you take those things into consideration, the results may not be as immediate as they would be if you came in and made a drastic pivotal style change on a specific product and a bigger ecosystem, but the longer-term value, like actually changing people’s habits and making the cultural shift, is what we think is more successful,” said Rodenbostel. “It’s better to keep in mind that DevOps is a journey and not a one-size-fits-all solution.”
Vladyslav Gram, head of DevOps at digital solutions company Ciklum, stressed the fact that the people affected by the shift need to be involved from the start.
“We need to make sure everyone follows [our DevOps] process. If someone doesn’t like it, if someone thinks there are problems with this process, we need to hear them and we need to fix their problems,” said Gram. “If we don’t have this type of integrity, then we don’t have a DevOps process at all.”
How to measure success
Continuous evaluation is essential for continuous improvement. However, it may not be clear how to measure the effectiveness of a DevOps culture, since culture is intangible. However, tangible metrics can provide insight.
Number of deployments. This is measured within a specific timeframe, such as the number of deployments per day.
Release frequency is an obvious one. Many teams have embraced DevOps with the goal of increasing delivery speed. However, faster delivery should not be achieved at the expense of quality.
Idea to production velocity. Is the time from business idea to production software decreasing? While this may be reflected in faster release velocity also, this metric considers the business aspect as well as the technical aspects.
Quality improvement. Quality improvement metrics include defect rates, percentage of successful builds, and percentage of successful deployments, all of which should improve over time.
Mean time to recovery. Infrastructure as code and autonomous databases are improving mean time to recovery. However, teams still need to track their progress here.
Business impact. While technical metrics can help explain the team’s productivity, speed, or quality improvements, ultimately, the purpose of DevOps is to speed time to value delivery.
“Profitability, customer satisfaction, cost reduction and risk reduction are the big four from a CFO and CEO perspective,” said Forrester principal analyst Charles Betz. “The thing I like about DevOps is a big step towards actually getting line of sight from technical operations to those business metrics.”
