In its 2015 report, the Open Web Application Security Project (OWASP) identified SQL injection and cross-site scripting among its Top 10 software vulnerabilities. Again.

If it feels as if you’ve been reading this same story for the last decade, it’s because you have. So why is it that we can build intelligent robots, fling unmanned vehicles throughout the universe and create computers that can recognize natural languages to a point where they can defeat humans on television quiz shows, but we can’t put an end to these software vulnerabilities?

John Steven, CTO of security company Cigital, said, “We’re in—and sick of—the hamster wheel of pain” that patching those vulnerabilities represents. “How many dynamic testing tools results parties do we have to have on every application before we get that there are certain truths that remain self-evident, like your hygiene around outputting coding and input validation continues to be bad?”

Rod Cope, CTO of Rogue Wave, agreed, saying “You would think after 15 years this wouldn’t be an issue. The OWASP Top 10 looks the same every year.”

Joseph Feiman, lead analyst for application security at Gartner, said, “Developers will keep developing insecure code, and there’s nothing they can do about it. It’s a losing battle with hackers.”

The life of software is so much different than it was just 20 years ago, when you built software, ran it behind a firewall, and left it unchanged until the next version was complete in six months to a year (or more). Today, according to Chris Raethke, CTO of testing organization Bugcrowd, applications are built horizontally using components. “All applications are pieces of the Internet,” he said, “dependent upon microservices and [service-oriented architecture].”

Further, said Raethke, developers are in a race against the clock as decisions regarding release dates, features and updates are driven more and more by non-technical people within an organization. “You can’t throw fairy dust in the air and get an app that’s secure, scalable and does everything [the organization] wants,” he said.

Whose responsibility is security?
For years, security advocates have said it’s up to developers to make sure the code they write is secure. Others insist developers don’t have the expertise in vulnerabilities and aren’t as up to date on known vulnerabilities as security practitioners.

“The bottom line is, we can’t make developers do this by hand,” added Cope. “It can’t be just another checkbox but a critical part of the [development] process.”

Enforcing security in code is anathema to developers, who are more interested in developing cool new functions than in ensuring the software is impenetrable. Cope said developers do not find security work interesting or fun, and they say to themselves, “I don’t get recognized for making my code secure. I only get recognized if I get a cool new feature out the door, or if we get hacked and something horrible happens, then I get blamed for it. But there’s no upside, other than avoiding more pain.”

Cope said another big problem facing development organizations is that as new developers come on board, they do not understand what weaknesses in code look like. “There’s always new developers coming on board, so the guy who got burned by [a hack] a few years ago and learned to avoid SQL injection issues going forward or buffer overflows, he’s maybe taking care of his code,” he said.

“But the guy out of school, or the person who just switched and is now writing code in a new language and doesn’t understand what that weakness looks like, is now introducing them into the code again. It’s not the same guy failing 15 years in a row, it’s a new crop of developers that are re-inventing those failures.”

More than not understanding code they didn’t write, developers coming in to an ongoing project need to understand what Gemalto product manager Todd Steel called the boundaries of the operating environment. Developer security testing has been “more about data object analysis and how to solve problems rather than understanding the system architecture and the limits of that system architecture.”

But Steel added that he believes strongly that “It is the developers’ responsibility to understand the environment and the vulnerability.”

Traditionally, developers write features and perform functional and regression tests, and then builds are made to ensure nothing is broken. Then, it’s up to the QA team to make sure it works properly. Cope said security testing comes at software from a different perspective. He said it’s the job of the security practitioner to look at how the software can be broken, not that it’s running properly. And these are skills developers often aren’t taught and don’t possess.

Steven said there will always be problems with the hygiene of a codebase. “We, the tester, we’re kind of like the dentist,” he said. “People go to the dentist and he asks, ‘Did you brush your teeth after every meal?’ And we’re like, ‘No, I had a business lunch and couldn’t.’ Hygiene is impossible. Asking developers to be hygienic as they write every line of code is impossible.”

Time for a new approach
Since securing software has failed in so many high-profile and damaging cases, the experts we spoke to for this article claim it’s time to reassess the way software is protected, through the use of automated tooling, or even the creation of a way for applications to protect themselves from hacks.

“There are things that are descriptive and discretionary, like password storage, which we know how to do right but is too hard for the garden-variety developer to get right,” Steven explained. “We can show them how to do it once. In an enterprise we can code it up for them once, and then they can use it and forget that they ever had that problem again…this notion of a security API, right?

“And then there are these hygienic problems,” he continued, “like the vulnerabilities that cause the injection problems we discussed earlier, that there’s no amount of testing or no single control that’s going to solve the problem. So we need to have the equivalent of the hygiene program, which is different libraries and frameworks. You use different tools to build the software that is hardened against those types of vulnerabilities so that it’s not possible for the developer to fall into bad hygiene or lack of hygiene that creates the vulnerabilities.”

Cope said you cannot expect developers to get security right by hand. “It’s just not going to happen,” he said. “This is just tedious overhead, as far as they’re concerned.

“So you’ve got to have some process in place as part of the development life cycle to make sure that it’s not just another thing you have to check off the box, but it’s a really important, critical part of the process. Just like you’d write code in an agile fashion and you would always have tests to prove that it works, your unit tests and functional tests and whatnot, I think security tests have to be elevated to that same level of first-class citizen.

“The second piece,” he added, “is you’ve got to have automation. You’re not going to get it right by hand the first time every time. Code reviews by hand are great, but not every line of code is going to get scrutinized properly. Developers are people… They get tired, they get hungry, they run out of caffeine at 3 o’clock in the morning, their eyes are getting blurry, you’re not going to have perfect code. You need tooling that can watch out for them all the time, whether they’re kind of asleep at the switch or not, or whether that’s important to them or not, some sort of tool can be reading that code and looking for security all the time. You’ve got to put both of these together, but you can’t really expect that developers are going to make this a top-of-line issue unless there’s some external factor requiring it. Management has got to elevate it.”

Gartner’s Feiman sees a new way forward. “If we cannot test all applications and cannot test them well, I see only one solution: Applications must test themselves,” he said. “We cannot diagnose all applications, that’s why apps must diagnose themselves. And because we cannot protect all applications with means that exist today—all these firewalls, IPSes, IDSes, Web application firewalls, encryption—the only solution is that applications must protect themselves.”

Software security is a gnarly hairball. As Feiman pointed out, “Software security is the most recent addition to the application security stack. Identity Access Management, 45 years old; network protection, 30 years old; endpoint, 25 years old. We are desperate because we see that attacks are absolutely pervasive and very successful, and we cannot stop them.

“But the network vendors keep selling them…keep pushing them, not just selling them. Same with firewall vendors. Same with endpoint protection. They keep adding new features, tweaking here and there, saying, well, now it will do much better, now it will do much better. And because we don’t know anything else, we’ll do it. So before penicillin, they recommended using some herbs, and you were taking them because there was nothing else in the market. What I’m saying is, that huge gap that had become clear to me four years ago when I starting working on that research, is to say, ‘How come, if we all claim that the app is the most precious asset—the app and the data it handles—how come that it’s absolutely powerless? It has no skills to detect, has no skills to protect itself. How come?”

And so, Feiman came up with the notion of Runtime Application Software Protection (RASP). But it comes with a caveat: “We gave identity and access management 45 years to evolve. And, after 45 years, it’s still not sufficient. We gave network security 30 years to evolve, and it’s not working. Well, it’s at the end of its capacities. We gave endpoint protection 25 years to evolve, and we still have viruses it cannot protect against.

“So all these technologies are complex implementations, and RASP is at the embryonic stage now, but there are already production implementations that have been acquired by enterprises and do a very important job of protecting the assets of the most famous organizations in the world.”

Another approach, suggested by Bugcrowd’s Raethke, is to create a “fire team” consisting of representatives of multiple departments that can champion change within an enterprise. “You’re taking one or two engineers, with an Ops guy, security, executive management, and one or two people from sales and one or two from marketing,” he said. This way, any feedback on the project is heard by all groups, and it’s an effective means of changing an operation. “This gives you a way to roll all your people through it.”

Communication across all organizational disciplines is key, said Raethke. “Any practice [including security] that you introduce to your organization has to be sustainable.” He suggested starting with two developers and two members of the operations team, then bringing in sales, marketing and executives after a commonality first been established.

Security training
Cigital’s Steven said that along with communication, training is critical. “We hire developers and train them on security because over that period of time we’ve not really succeeded in doing the opposite,” he said. “I think when you look at the security practitioner community, a lot of them are testers and security practitioners that are beginning to learn a bit about coding to be more effective at their job, and frankly, I think a lot of this design—we’re talking about really design decisions, talking about classifying the problems by the kind of flaw, and taking the right kind of design or software security initiative approach to solve it—I think it’s a little bit beyond their skill set.

“There are different classes of people,” he continued. “We’re starting to see from the OWASP community these testing practitioners: Some of them have five years’ experience, 10 years’ experience now, doing penetration testing. This group of people may not know development, they may not have really talked to executive management before, so the politics in building a security group may be a challenge. When they stroll down to talk to the architects in the organization about adopting a new open-source framework or library you’re going to build your website on, you can imagine that’s going to be a tough climb for them. There’s a development gap there, and an architecture gap.”

Training and planning for vulnerabilities are tips that Rogue Wave’s Cope offered. “The only way to really protect yourself there is is to stay up to date on the latest patches, latest news, latest fixes, and expect them to continue… With all software, there will be more security holes, you need to plan for it, have tooling, prepare for some notification process so you can quickly learn when there is an issue, whether it’s open source or from somewhere else, that you know there’s an issue, and then have a mitigation plan in place so you know what is affected.

“If there’s a new OpenSSL patch that comes out, well, what do I do with that? How do I know which machines in my environment—either virtual or physical—need to be updated? And how do I do it? Who’s going to do it? The whole mitigation plan needs to be an ongoing, long-term effort.”

Fighting the good fight
All agree that as long as there is software, people will look to exploit weakness for whatever nefarious reason. But just because hacking ultimately can’t be stopped doesn’t mean it’s not worth the effort to try to secure software.

Rogue Wave’s Cope put it this way:

“It’s a little bit Darwinian…survival of the fittest. If you are patching these things as fast as you can in your enterprise, you’re going to fend off the hackers that are kind of at the bottom of the hacking food chain, the script kiddies and things like that. They know some basic techniques, maybe they’re old techniques even, but they still work on sites that aren’t being updated and patched. So if you’re doing your job as an enterprise and you’re applying with diligence and mitigating your risks, you’re at least going to cover the older known vulnerabilities, whereas the guy down the street who’s not doing that is going to be an easier target, and therefore the hacker that spends an hour trying to get into your site might find there’s easier pickings down the street. He’s feasting on the next guy and leaving you alone.

“It’s unfortunate, but you’re really not necessarily competing against the hackers; you’re competing against the other guys who aren’t keeping up their defenses as fast as you are. Kind of like, you need to put on your tennis shoes, not that you’re going to outrun the bear, but you’re going to outrun your friend.”