Software is the lifeblood of most businesses today. So, what happens if that software is unreliable or insecure? It seems like a no-brainer that the software being pushed out should be protected. But, as software is being developed and deployed at a rapid pace, an important aspect of the life cycle gets lost in the race: Security.
Security is not only important because it can help prevent hack attacks and data breaches, it is important because one mistake could have a significant impact on a business’ reputation and sales, according to Walter Capitani, product manager at Rogue Wave Software, a software development company. “Customers want to use new features and upgrades more quickly, and aren’t willing to wait on development,” he said. “The unpredictable nature of fixing security issues means that release dates keep changing, which frustrates customers and delays new customers from purchasing the product.”
The problem is that organizations are so hyperfocused at releasing software faster to stay ahead of the competition that they adopt DevOps approaches to modernize their strategies, but still practice legacy security approaches, according to Peter Chestna, director for developer engagement for Veracode, an application security company.
“The biggest problem today with application security is that the development organization is not goaled to secure their software. They are goaled to release software quickly. Without a mandate and shared accountability between security and development that is measured and reported at every level of the organization, security will continue to be hard,” he said.
Legacy approaches are no longer an acceptable method to delivering secure code today, Capitani explained.
DevOps has proven it can help speed up the development process, but in order for businesses to really take their development processes to the next level and keep up with the next generation of innovation, DevOps teams need to figure out how to take security information or security intelligence, relate it to code, and deliver that to teams as early and as often as possible so they can make the right decisions, according to Derek Weeks, vice president and DevOps advocate for Sonatype, a DevOps automation provider.
DevOps, at a high level, is about making the development process easier through automation and eliminating risks. If DevOps is enhanced with tooling and strategies that make it a lot more proactive in finding vulnerabilities, identifying bugs and fixing them right away in real time, then that is going to bring your security risk significantly down, according to Srini Vemula, technical program manager for SenecaGlobal, a global technology consulting firm.
This enhanced DevOps approach is being called DevSecOps. “DevSecOps seeks to bring security to the table to be involved and integrated into the DevOps team and their responsibilities. By shifting security left, the team de-risks their software by finding and fixing security vulnerabilities early in the SDLC, usually before check-in,” said Chestna.
A DevSecOps approach still provides the benefits of DevOps — developing, deploying and delivering new features to customers fast — while removing an antiquated security process, explained Chris McFadden vice president of engineering and operations at SparkPost, an email delivery service for developers and enterprises. “We can’t afford to be slow, we have to be fast,” he said. “You can no longer treat security as some other team out there that it is a nuisance. You have to really build that into your software development process.”
According to Ian Head, research director at Gartner, a research and advisory firm, DevSecOps is part of a journey to a “mode 2” style of working. “Mode 2” means the business is exploring and experimenting with solving new problems and optimized for areas of uncertainty. “Mode 1,” according to Head, was about predictability, improvement, and well-understood areas. To reach a “mode 2,” new and innovative approach, the entire organization needs to change, and everyone needs to embrace it in order to be successful, Head explained.
Culture remains a problem
When DevOps first emerged, one of the biggest problems organizations faced when transitioning was trying to break down silos. Historically, the development and operation teams did not work well together because their goals were not aligned. Developers had the mindset to bring change, and the operators or the IT team feared what that change would do to the reliability of the solution. But together, they became this DevOps powerhouse that released software faster, with better quality.
Today, the same siloed problem is happening when you try to transition to a DevSecOps approach. Getting teams to be on the same page and collaborate is a tough process no matter what departments you are trying to bring together, SparkPost’s McFadden explained. What it comes down it is getting people and leadership oriented towards the same goals and objectives.
“You can have the greatest plan and strategy in the world, but if you have a culture that isn’t dedicated to securing the development process around DevOps, you are going to run into a problem,” said Flint Brenton, CEO and president at CollabNet, a DevOps and agile solutions company.
In a traditional operations world, the security team sat the furthest away from the developers, while the testers and operators sat the closest. According to Sonatype’s Weeks, this type of structuring is one of the main reasons security has become an afterthought for DevOps teams. “Developers weren’t necessarily trained on security. It wasn’t a natural behavior for them or a part of their training. Just because security sat further away, it makes it that much harder to bring it in from a culture of organization perspective,” he said.
The security team has to be brought into the DevOps teams so they can understand the code and work directly with the developer sooner so they know what the application is suppose to do and not supposed to do. “The strategy starts with people. Security and development must build a relationship with one another. There should be discussions about struggles and goals to build mutual empathy and understanding. At the highest level of the organization, security and development leaders should agree on mutual goals for the security of the software that is built,” said Veracode’s Chestna.
The company also needs to train its development teams in security. According to Gartner’s Head, developers don’t have to become security consultants, but they should know basic security skills so that they can be trusted to follow good security practices.
In addition, the security team needs to adopt new tools that promote collaboration and provide security analytics. They can no longer afford to work in their old, siloed, and security-specific tools, according George Gerchow, vice president of security and compliance for Sumo Logic, a cloud-based log management and analytics service provider.
“Security only seems hard because we are untrained. With the proper training and tooling, developers can learn to write software securely the first time. That will prevent unplanned work farther downstream and avoid the costs of finding, tracking, fixing, testing and verifying the changes necessary to pass policy. In DevOps, it’s all about failing fast and verifying quality (including security) at the earliest possible time. A secure mindset can be learned and secure coding can become second nature,” said Chestna.
Implementing a DevSecOps strategy
Once the culture is established among the DevSecOps teams, there are additional methods and tactics that can add to the success of DevSecOps.
The number one best practice for implementing DevSecOps is to simply detect vulnerabilities as early as possible in the software development process, Rogue Wave’s Capitani said. Teams can do this by applying techniques such as static code analysis, “which can discover vulnerabilities even before code is compiled for the first time, and reduce the time associated with finding and fixing security vulnerabilities,” he said.
At Sumo Logic, the code analysis is one of six pillars it enforces in a DevSecOps strategy. The other five are: change management, compliance monitoring, threat investigation, vulnerability checks, and security training for developers.
Code analysis rides along with building code into small chunks, and doing multiple releases a day. “This allows us to start looking into our continuous development chain very rapidly to make sure we are not releasing things that have vulnerabilities in them,” said Sumo Logic’s Gerchow.
Change management at Sumo Logic empowers all of the development teams to implement changes, and have them implemented within 24 hours. The teams work through Slack and JIRA to make changes quickly, and determine if a change is going to be good or bad.
Compliance monitoring helps gather evidence for compliance while the code is being developed in order to shorten the auditing process.
Threat investigation is where the team discovers and remediates any threats across any of their services.
Vulnerability checks ensures the team is constantly scanning all of its code and environments to look for security threats and remediate.
Training developers goes beyond teaching them OWASP or on-site solutions. Developers need to go to hacking events and see first-hand what real-life hacking looks like. This reminds them why they need to do all of the above checks.
SenecaGlobal’s Vemula added that a mature organization will define its security architecture through CIA, or confidentiality, integrity and availability. It is about getting these three tenets of security implemented across the entire organization without becoming an impediment in terms of agility.
Vemula explained security is often short-changed because developers don’t want to have to regularly change passwords, or apply automatic processes to apply patches. Having a well-defined security architecture and principles, and being able to apply them to DevOps, will help teams be in control of what is going on in the development life cycle.
In order to know whether an organization is on the right track, it needs to ask key questions like: Do we know when we are compromised? What is our ability to respond to the compromise? How confident can we say that all the software and third party services used today in production are well patched and running the latest versions as possible? “This type of questioning will enable teams to identify a lot of weak areas,” Vemula said.
Another sought-after, but hard to find, strategy or role on a DevSecOps team is the role of a DevSecOps specialist, according to CollabNet’s Brenton. A DevSecOps specialist is constantly examining every step in the development process, and making sure the team is focused on securing code. “Having a DevSecOps security specialist who is constantly analyzing what you are doing, comparing it to market trends, comparing it to best in class, and then coming back and telling you not only what you can do better, but how you can do it, is an enormously valuable position to have in the enterprise, and folks that have this degree of speciality are extremely valuable,” said Brenton. However, today it is a bit unrealistic to have one person responsible for all of this. Instead, it should be a team that has experience in security and DevOps, and can provide a point of view from what works in other companies as well as what worked in their own experience. A way to add this type of specialist or team into DevSecOps is by growing them internally within the company, or bringing on a consulting firm that specializes in this area and can constantly educate you, Brenton explained.
In addition, Sonatype’s Weeks said DevSecOps is as much of a tooling problem as it is a culture problem because siloed teams normally work in their own specific set of tools. When DevSecOps is shifting security left in the development cycle, it introduces a need to have more decision making, more evaluation-type criteria, and more quality or performance evaluations as early as possible in the development life cycle.
Tools should include automated tests early on and as early as possible, intelligence to analyze the code for known vulnerabilities, and be able to monitor code in production for any red flags. Vemula adds that there is an umbrella of tools available on the market that enable continuous vulnerability assistance all while integrating well within a DevOps initiative; it is just about finding the ones that will enable teams to apply application security best practices and check for vulnerabilities before they are put into deployment.
“The more precise a security tool is in the DevSecOps environment, the more efficient it is going to be at doing tasks. If you are getting good and bad results that aren’t accurate, you actually are not that much faster. You are getting the answer back faster, but if the answer is wrong you are actually inefficient,” Weeks said.
In addition, the tools need to be highly scalable so that features such as vulnerability testing can be extended as the team grows, as well as be easily integrated into CI systems, Capitani explained.
At the end of the day, Gartner’s Head says DevSecOps is more about continuous security or continuous assurance. “We are much more focused on continual learning and continual improvement rather than the perfect projects and delivery,” he said.
DevSecOps guides and resources
Introducing security champions to the DevSecOps life cycle
One of Gartner’s top 10 steps to a successful DevSecOps approach is to “Adopt a security champion model and implement a simple security requirements gathering tool.” According to Synopsys, developers make great security champions because they are familiar with the organization’s software and development groups, and have a deep understanding of technical issues and challenges their organization faces. This Synopsys white paper walks organizations through recruiting software developers as security champions, and injecting security all throughout the life cycle.
Necessity is the mother of the ‘Rugged DevOps’ movement
Tools are an important part of bringing development, operations and security under one umbrella. This DevOps Buyers Guide provides a guide to DevOps offerings, and how tools can help teams consider security as a first-class citizen in DevOps. “Security is still one of the last places where that archaic approach of development handing off the software to a different team and walking away still reigns,” Tim Buntel, vice president of products for XebiaLabs, said in the guide. “Secure software is just good software, and good software is secure software. Everything that we’re doing in DevOps is allowing us to build better software at scale and release it faster.”
Automating Security in DevOps – Security in the Pipeline
In this video, DJ Schleen, information security advisor at Aetna, talks about how organizations and DevOps teams can implement security controls into continuous delivery pipelines. According to Schleen, security controls can help promote secure code to production and provide minimal impacts. Schleen also walks through securing applications from throughout the application, from conception to deployment, as well as how Aetna integrated security into its lifecycle. Security for DevOps and DevOps for Security is an “unprecedented opportunity. It allows us to disrupt traditional mindsets of security,” Schleen said. “It gives us an opportunity to collaborate with these folks and really improve the culture of the organizations that we work in. Security is just a part of everything we do on a day to day basis.”
Governance and Transparency in GovSec DevOps
Leonel Garciga, chief and CTO of the Joint Improvised Threat Defeat Organization, recently gave a keynote at the All Day DevOps conference in November to talk about the government, DevOps and how CISOs and CIOs can approach this new strategy. According to Garciga, having a DevOps pipeline is imperative for CIOs and CISOs because it provides a single pane of glass for delivery and risk assessment; bakes in regulator and auditing functions; provides real time visibility into assessing risk and continuous monitoring; and provides a repeatable process with metrics.
Hard Won Lessons of a DevOps Addict
“We have to be 100% right. Attackers only have to be right once. And that means the odds are stacked against us,” Shannon Lietz, the director of DevSecOps at Intuit, recently said at the Lonestar Application Security Conference in the beginning of the year. She talks about the rise of DevOps and how security needs to evolve. “We could change the world if we got security to be everybody’s responsibility, and it is our job to make that happen,” she said. “What you will get out of DevSecOps is safer software sooner.”
