The DevOps community is all about speed and high-quality releases, and since mobile applications and websites drive traffic and revenue, companies can’t afford to have site crashes or send out buggy releases. To emphasize the importance of testing, people are putting it back into DevOps, literally, so it shifts both left and right to the development and the operations team. Adding testing in the middle can allow teams to maintain velocity, while also making sure sites are up and running and the releases are still high quality.

The digital evolution has transformed how businesses generate revenue, and companies are well aware that testing has been one of the most important aspects of software development. Dev-Test-Ops might just be another trendy buzzword, but it’s also helping testing finally come into focus. For software development, quality matters, and the only way to achieve that is to continuously test throughout all stages of the process, and to make sure a testing strategy is in place to conquer all testing challenges.

The fear of DevOps
When DevOps first became a hot topic, no one knew exactly what it meant. Organizations finally started to understand that with DevOps comes speed, building applications quicker, and adding new features or capabilities to compete and drive revenue or traffic. One aspect in DevOps that still gave companies trouble was testing.

In the past, CIOs and developer managers had to make a choice between meeting a deadline, or delaying the delivery and running a few more tests just to be safe. Testing started to be deemphasized when many chose the speed element over adding more testing before delivery, according to Tom Lounibos, CEO of SOASTA.

(Related: DevOps jobs are among the highest paid)

This was before websites and mobile apps became revenue producers, he said. Now, testing is changing significantly because it is no longer something that is just done prior to a release—it has moved into production as well. The old-school view of DevOps reducing the need to test isn’t accurate anymore, and most companies realize that full automation and testing a hundred times more than before is the best way to test.

That’s why it’s important now more than ever to understand continuous testing, according to Jason Hammon, director of product management at TechExcel, because companies are going to have to make sure their releases are high quality and delivered in a very limited time frame.

“[DevOps] couldn’t possibly reduce the need for testing,” said Hammon. “It reduces the time for testing for sure, but I think if anything it makes the need for testing more important because with the Continuous Integration strategy or DevOps strategy, you are going to be deploying code faster.”

DevOps might have been threatening as a global trend, according to Jonathan Bracken, vice president of marketing at Neotys, and it never was a world testers could fully embrace because the concept of not testing wasn’t something they could live with. Now, including testing in the approach of going more quickly and in a more automated way through the application development life cycle makes a lot of sense, according to him.

Why Dev-Test-Ops?
Testing is inevitable, so why create another term to drive home its importance? Dan McFall, Mobile Labs’ vice president of mobility solutions, said that he thinks a phrase like Dev-Test-Ops is needed because some developers still consider unit testing to be enough. It can be argued that unit testing is enough since it can catch potential bugs or regressions, but testers still need an overarching testing strategy that involves integration testing and non-functional testing and professionals trying to break to break the software, he said.

Alon Girmonsky, CEO of BlazeMeter, said people will always try to coin a term that will better describe the current state of the software delivery process. But Dev-Test-Ops is relevant because it shows how today people are understanding that testing has become so critical, and that it’s not just an afterthought.

“The one ingredient that glues every step together is testing,” said Girmonsky. “So think about it: When you write your code and you commit your code now as opposed to five year ago, you test it. It’s being tested automatically behind the scenes, and only if the test passes, only then does it move to the next stage.”

Everyone has their own definition of these terms like DevOps, and according to Rod Cope, CTO of Rogue Wave Software, no matter how you describe it, it’s all about writing code to automate the things that use to be done by hand.

“By saying Dev-Test-Ops, we are kind of recognizing that test is also something that used to always be done by hand,” said Cope. “Now there has been further effort into developers automating tests as well, so now Dev-Test-Ops kind of points out that developers are leading the process all the way through from the first line of code to the actual functional, performance and unit testing—all the way to deploying, getting infrastructure up and running in production, and then testing that production site before it’s made open to the public.”

Dev-Test-Ops also gives companies the opportunity to remind the marketplace that testing is important, according to Matt Brayley-Berger, worldwide product marketing manager for HPE. Those who are trying to release faster but aren’t integrating things like quality or security testing are just going to get bitten in the end, he said.

One thing that companies should keep in mind is that DevOps is not the destination, it’s actually just the beginning, according to SOASTA’s Lounibos. He said that companies are going to do process compression for development, testing, operations and monitoring, and as businesses get more involved in this process compression, “You are going to see it inform the developers, inform the testers, inform the operations guys, and the business folks more and more as we go forward.”

Tackling testing challenges
As organizations adopt continuous testing and automation tools, challenges are hard to avoid. Testers need to be more comfortable, as well as more technical, said Brayley-Berger. They will need to play a role in planning for automation, and it’s too difficult to do that with manual testing, according to him.

“It is really getting that tester to become more comfortable with the automation,” said Brayley-Berger. “There is some tooling that can assist with that, and the reality is—and this is from our perspective as a vendor—tools are only part of the solution.”

Developers are focused on developing, and Ops teams are focused on deployment. But in Dev-Test-Ops, both need to do adequate testing. Another challenge that can appear in any organization, according to Girmonsky, is the coverage challenge. Teams need to do testing in order to cover all of the risk aspects of the software, but due to time constraints and resources, it can be tough to achieve that high-level of coverage.

A lot of the traditional challenges around just automating tests still exist, according to Cope. He said it’s not easy to perfectly mimic production data, especially when dealing with Big Data and needing a lot of it to understand how a system performs under load. Another challenge is just trying to get the full production environment itself up and running, but Cope said tools like virtual machines can help with that.

Other challenges come with testing for mobile, and the real issue here is with the device component, according to McFall. He said that most recommend having a real device component because there are “bugs that just do not show up on emulators, that will consistently show up on real devices.”

The role of the right tools
The No. 1 tool companies need if they want a Dev-Test-Ops environment is some sort of automation tool. Historically tools have been siloed, but today’s tools need to be automated and integrated into the Continuous Integration system.

Lounibos said that the technology needs to start with an automation tool at its core so that teams can generate tests and have the ability to analyze results like performance analytics specifically.

“It’s very much an important tool, but performance analytics is becoming a data-driven environment,” said Lounibos. “You don’t just want to build something without [an] understanding of who your user community is going to be, and you don’t want to just test something that you think might be the way users would use it. You want to actually extract user information to determine how to test going forward.”

Be smarter about how you use automation tools as well, according to Hammon, who said to consider automating areas that are stable so you can use automation for smoke-testing some of the builds, and then use the little time you have left to focus on manual testing to make sure the functionality is tested.

Hammon also said having a good test-management solution can track some of the overall coverage. A tool that can see what has been done and what needs to be done (or what areas are low risk or stable) can help a team identify what areas are yielding the most problems.

The minute an organization decides to introduce Dev-Test-Ops, they need to say that they are investing in the automation procedures, according to Girmonsky. Increase the coverage and attempt to have a predictable time to release, and the Dev-Test-Ops solution will help overcome a lot of these testing challenges.

Girnonsky urges any Dev-Test-Ops adopters to consider open-source tools, as they have been built in the “era of automation.” He also said another component that can contribute significantly is the cloud, because you can run any number of tests in parallel and keep on adding tests to increase coverage.

Tooling is only half of the solution, and beside the right automation technology, companies should consider the monitoring components as well. Not only are organizations going to need to test their apps using a certain set of assets or tools, but they are then going to need to monitor it, according to Neoty’s Bracken.

And, for the tools and monitoring components to become enablers of a complex process shift as (Bracken puts it), they will need to be automated and integrated.

“If your tools aren’t integrated with your Continuous Integration system, it’s going to be a lot harder to fit performance testing into the process,” said Bracken.

Performance testing in Dev-Test-Ops
For performance-critical applications, considering how and what is being tested throughout the entire life cycle is key. Organizations need to figure out the appropriate process for them so they can add value to the activity of performance testing and validating it through every stage of the application life cycle, according to Bracken. He said that some of the major challenges of performance testing in a DevOps world include the following:

With DevOps, speed is king: Most performance testing tools/processes can’t keep up, which is why performance testing often becomes an afterthought in a DevOps environment.

Analyzing test results: Performance test analysis usually takes time. In a DevOps environment where all other test results are analyzed automatically, performance testing might easily be skipped if the results aren’t provided as pass/fail.

Speaking the same language between QA and Ops: QA will test specific test cases, Ops uses unrelated tools to monitor the production environment. If something goes wrong in production, and QA can’t reproduce the issue, the same SLAs and use cases need to be tested and monitored through the process to ensure performance is validated at every step.

See here for more in-depth analysis from those interviewed in this story, and here for a roundup for Dev-Test-Ops offerings.