DevOps and security teams are learning how to work together, albeit somewhat awkwardly in these early days of DevSecOps. One reason why it can be difficult to get the partnership “right” is that people define DevSecOps in different ways.

“If you asked a room of 10 people to define DevSecOps, you’d get 15 definitions. I think we’re starting to coalesce, but I still think there are a lot of misconceptions and there are still a lot of people who are going about it the wrong way,” said Caleb Queern, principal at KPMG Cyber Security.

One challenge is approaching DevSecOps from a tools-only perspective, without considering the impacts of change on processes and people.

RELATED CONTENT: 
Creating a DevOps culture
CI/CD pipelines are expanding

“A lot of buyers will procure something, put it in place in their organization and declare victory,” said Queern. “It doesn’t lead to the business outcomes that most folks are looking for.”

In a December 2019 report, Gartner estimated that by 2021, DevSecOps practices will be embedded in 60% of rapid development teams, as opposed to 20% in 2019. To succeed, DevOps teams must prioritize security and security teams must adapt to the accelerating pace of development.

Is shifting left enough?
DevSecOps reduces the number of application vulnerabilities, but shifting left is not enough.

“Shift left has truly moved the needle on capturing a lot of these early-stage, often ignored, vulnerabilities [but] it doesn’t necessarily solve all of the security challenges for an organization,” said Jake King, co-founder and CEO of Linux security platform provider Cmd. “We have to be vigilant, we have to be aware, we have to be continuously testing.”

Application exploits are the most common way hackers infiltrate organizations. So, in addition to running a Common Vulnerabilities and Exposures (CVE) scan late in the SDLC, shifting security left adds another security point. However, security should be integrated into every aspect of the application life cycle because as a recent Trend Micro report states, “Attackers will find ways to take advantage of any weak link to compromise the DevOps pipeline.”

“[Application] security has kind of been looking at the trees and missing [the] forest,” said Zane Lackey, former chief information security officer (CISO) at ecommerce marketplace Etsy. “Trying to sprinkle a little bit of security dust at the end or at the beginning [means] you’re missing the forest, which is that you need to be empowering teams to think about and have security capabilities from when they’re designing new services to when they’re implementing them to when they’re actually deploying and running them in production.”

Sandy Carielli, principal analyst at Forrester Research, agrees.

“We need to make sure we’re not just addressing security in the design and development phase, but also in the testing and deployment phases,” said Carielli. “It absolutely needs to be addressed throughout the life cycle. The vulnerability is going to survive for a much shorter lifetime if there is regular scanning.”

End-to-end application security results in more secure code. And, full life cycle visibility into exploits helps teams understand how and where their applications are being compromised.

“The way security was done historically frustrates developers. You wait until the very end and it takes a week for them to get back results. And a lot of the tools that we’ve used historically, things like static analysis, create an unacceptable [number] of false positives,” said Neil MacDonald, vice president, distinguished analyst and Gartner fellow emeritus in Gartner Research. “It’s the antithesis of what we want for DevOps. We’re supposed to be enabling speed and agility and so security has to shift left, integrate natively into the developer’s environment and catch these problems as soon as possible so the developer isn’t slowed down at the end.”

However, baking security into the entire SDLC takes time.

“This isn’t something that you finish, ever,” said KPMG’s Queern. For one thing, black hat tactics are constantly evolving.

It’s also important to pay attention to access rights.

“It’s not so much about technology components, but how you’re divvying up who does what,” said Fernando Montenegro, principal analyst, information security at 451 Research. “If I have an artifact here, am I allowed to interact with it? Am I allowed to do something with it? Do I have the right credentials to do dev versus QA testing versus production etc.? And then at runtime, is this thing behaving the way it’s supposed to behave and if something goes wrong? Do I have the infrastructure to understand what happened? You absolutely need that across the stack.”

What the DevSecOps team should look like
Moving from DevOps to DevSecOps isn’t a matter of putting a security team member on a DevOps team, because developers tend to outnumber security professionals by 100 to one. Since most developers aren’t security experts and most security experts aren’t developers, the two must learn to work together effectively.

“Research shows that the places where we talk more about security, we’re better at security,” said Queern. 

Queern and others interviewed for this story underscored the importance of “security champions” who serve as the security expert on a DevSecOps team. Security champions are usually a developer who volunteers for the role. In Queern’s experience, the security champion participates in labs, demos or presentations that focus on security and application development best practices. While the application security team may host the trainings, the presentation duties are very quickly assigned to the security champions who will serve as security experts for their teams.

Health management platform provider Advantasure has a security liaison who works across more than 500 developers, according to Chief Information Security Officer (CISO) Wallace Dalrymple. The security liaison is responsible for upskilling the developers who will become security champions. Specifically, the liaison oversees their e-learning training programs and is the first escalation point when a security-related question or issue arises.

The ROI on training dollars tends to be poor if the person attending training doesn’t apply the newly acquired knowledge on a regular basis, however.

“The best type of training is training in the moment. Having that training built into the development process a little more closely will allow the teams to start absorbing it and living it a lot more,” said Forrester’s Carielli. 

Charles Betz, global DevOps lead serving infrastructure and operations professionals at Forrester, thinks it’s important to provide DevSecOps teams with secure baseline infrastructure environments.

“In many cases, people just don’t know what needs to be secured,” said Betz. “So, give them an image that’s secured, give them an environment that’s secure, give me acceptable patterns for using the database, using Apache Kafka, using a load balancer in the firewall. Then, have them focused on what they need to be focused on.”

What the relationship with the security team should look like
There’s a diversion of opinion about who should initiate the DevOps-security relationship. Should DevOps reach out to security or should security reach out to DevOps? There has been some friction in the relationship because security has served as a barrier to faster application releases.

“There’s no time for old school security testing every release. Fundamentally, security should be something that’s just like quality. It’s one of the fundamental requirements of your code,” said 451 Research’s Fernando Montenegro. “The role of security is collaborating with that development team to let them know what it is that they need to do. Here are our corporate objectives around security and the kinds of things that we want you to be aware of. Here’s how you threat-model your application to define what needs to happen where and then let the development team run with it.”

Traditionally, security’s first instinct is to prevent developers from introducing risks in the first place. For example, Etsy’s former CISO Zane Lackey recently met with the CSO and CIO of a Fortune 100 company separately, about 30 minutes apart. When he asked each of them how they think about cloud and DevOps, the CSO said he wasn’t allowing them to happen because they’re not secure. The CIO subsequently said they had 50 cloud apps now. By the end of the year, they’ll have 200 and in 2021, a couple thousand.

“I looked at him kind of funny, and he said, ‘Oh, you just met with the CSO,’ ” said Lackey. “We don’t tell them [anything] anymore because they try to say no to everything. If they want to enable us, then we’ll incorporate that, but if they’re just going to be gatekeepers and say no, we’re not going to speak to them anymore.”

Good progress is getting security and DevOps working together. Better progress is when security and DevOps cooperatively integrate security practices throughout the SDLC in a manner which doesn’t create friction in the DevOps pipeline. 

Supply chain security matters
Developers are using more open-source and third-party elements in their code than ever before to meet time-to-market mandates and focus on what matters most, such as competitively differentiating their applications. However, Gartner estimates that fewer than 30% of enterprise DevSecOps initiatives had incorporated automated security vulnerability and configuration scanning for open source and commercial packages in 2019. That number is expected to rise to more than 70% in 2023. 

“Security needs to go all the way back to the point where they’re downloading code from GitHub. Let’s tell them this is a known vulnerable component. We’re not going to download that because it contains a known set of critical vulnerabilities,” said Gartner’s MacDonald. “Let’s tell them that before they download it, before they stitch it together in their application as they’re writing code in their IDE.”

Advantasure’s decision to implement application security platform Veracode was made jointly between security and DevOps.

“We can’t dictate to the business, you must do it this way,” said Advantasure’s Dalrymple. “It was really us reaching out to them and saying you’re going down this path. How can we be leveraged in your environment without having to hire an army of application security architects just to support you?”

Measuring success
It’s hard to tell how effective DevSecOps is if it isn’t being measured. There are some of the classic methods like bug tracking and code coverage, although Forrester’s Betz said it’s important to start with the target operating condition.

“What is your risk tolerance? What are your metrics there? What are some of the desired metrics with things like code coverage and what level of assurance do you require in your systems?” said Betz. “Really look at your highest governance metrics that drive your security posture and how you understand your attack surface. Those drive your operational processes and procedures which will include things like software, bill of materials, supply chain, vulnerability scanning, pen testing and red team testing.”

Cmd’s King said time to detection and resolution of an infrastructure issue is important, as is code coverage and code hardening. Measuring customer perception is also wise because security is a trust issue.

“If you keep on top of it, if you incrementally add security, you bake it in from the start and add it early in your processes, you will see dividends paid up majorly in the long term because you will not be dealing with legacy,” said King.

Another metric has to do with the collaboration between DevOps and security.

“A common metric is the number of bugs and if they’re going down over time. This has probably done more disservice to application security programs than anything else because there’s a million questions within that that don’t get answered,” said Lackey. “The healthiest metric you can track is how often DevOps teams proactively interact with the security team because if those teams are proactively coming to you and focusing on that metric, whenever they think of new technologies or architectures you’re going to be able to reduce risk.”

Teams also should be counting security incidents such as breaches and data leakages, insider and external exploits and advanced persistent threats.

“[Whichever way] you’re counting them you want to be baselining and continuously improving,” said Betz. “I strongly believe in benchmarking against yourself.”