In a time when software must be released on shorter and shorter cycles, there needs to be a way to make every step of the process faster. No longer can developers build and then wait until they are finished to test everything. In order to stay ahead in this digital world, organizations must look to continuous testing.
Continuous testing can be defined as testing that happens continuously and automatically as code is developed, said Jeff Scheaffer, general manager of continuous delivery at CA Technologies.
Gerd Weishaar, chief product officer at Tricentis, defines continuous testing as “the steps that are necessary to actually make sure the quality of the software that is being developed is ensured continuously.” He says, “you’re in lockstep with the development cycle and you are continuously ensuring the quality of the software that has been developed is okay.”
“Strictly speaking, continuous testing is the practice of introducing automated testing into the entire software delivery pipeline gaining faster quality feedback throughout the whole release,” said Sean Hamawi, chief technology officer and co-founder at Plutora.
According to Hamawi, continuous testing can be comprised of unit tests, integration tests, and end-to-end tests. Unit tests are part of the check-in process when the continuous integration server is notified of changes. These tests are sometimes done on a 15 or 30 minute cycle. Integration tests are regularly run, often nightly, throughout the development cycle, often for several hours at a time. The end-to-end tests are planned on a longer cycle, usually weekly or bi-weekly. These tests usually include a broad set of automated tests. The depth of inspection increases as the code moves closer to production, according to the company.
RELATED CONTENT: A guide to Continuous Testing tools
“For me, continuous testing is a key and critical component for a DevOps initiative,” said Mark Lambert, vice president of products at Parasoft. “DevOps is all about removing the bottlenecks in your software development process or your software delivery process so that you can realize the benefits of Agile. If you have a lot of manual steps between writing the code and actually getting something out there, then you really don’t get the benefits of Agile because you end up spending all of your time doing the release and deployment component and not enough of the time doing the development.” By properly implementing continuous testing practices, the benefits of Agile can be realized.
“The promise of Agile to the business is that you’re going to be able to deliver more to the market and deliver it faster, but without continuous testing in place you really can’t accelerate without sacrificing quality,” said Lambert.
The interest in continuous testing has continued to increase in recent years. “What’s going on in most of the industries is what’s called a digital transformation,” said Weishaar. “And a digital transformation means that you have to serve your customers faster and in a better way than before. Otherwise the competition is going to disrupt you.” Companies that understand this will be able to put out releases faster and stay ahead of their competition.
“Time pressure reigns in software development, and now that it’s possible for organizations to ship software at will, businesses demand more from development teams, and faster,” said Philip Soffer, CEO of test IO. “Continuous testing is an outgrowth of the Agile and DevOps movements and holds out the promise that organizations can shift faster while simultaneously improving quality.”
“Ultimately, testing is the ‘final frontier’ of sorts in terms of improving the overall software development life cycle,” said Scheaffer. “Approaches like Agile and DevOps have helped accelerate both the pace and quality of software that’s delivered, but there is an increasing recognition that testing processes have to continue to evolve to keep pace with advances in other areas.”
According to a study conducted by Freeform Dynamics that CA Technologies sponsored, continuous testing leaders are 2.4 times more confident in software quality, 1.5 times more likely to deliver 10 times faster, and 2.6 times more likely to reduce costs by 50 percent. With shown benefits such as these, it is no wonder that continuous testing has become so popular.
Scheaffer lists speed, quality, and costs as the three primary benefits of continuous testing. According to them, these three things lead to a better user experience, which then leads to increased customer loyalty and satisfaction. Those using CA Technologies’ continuous testing tools have experienced 50 percent faster delivery speeds, improved test coverage by 80 percent, and reduced testing costs by 25 to 85 percent, according to the company.
“When adopted correctly, continuous testing centers testing in the software development process, creating a constant rhythm of refinement and a culture of collaboration throughout the software development life cycle,” said Soffer.
“Continuous Testing strategies look to achieve four capabilities: Test early, test faster, test often and automate more tests,” said Hamawi. “Earlier testing reduces the overall cost of fixes. Faster testing improves the efficiency of the development machine. More frequent testing improves the overall coverage.” By testing earlier in the development cycle, bugs can be found sooner and fixed faster.
“The benefit of continuous testing really is to enable the promise of Agile,” said Lambert. In order to truly have an agile development lifecycle, organizations must adopt continuous testing. If they do not, they will be wasting both time and resources saving testing for the end.
According to Hamawi, there are a number of challenges that can be faced when switching to a continuous testing model. It can be difficult to discover test coverage, which is typically discovered by creating a requirements traceability matrix. He recommends that organizations include both automated and manual tests in the traceability matrix.
Usually developers provide test cases for positive and happy paths, but with continuous testing there also needs to be units tests for negative paths, boundary conditions, injection, and data conditions, to name a few, says Hamawi.
Another challenge noted by Hamawi is the fact that the business, testers, developers, and operations are all working off of different sets of requirements. As a result, each feature will go through an evolution between what the customer wanted and what was produced.
The final challenge that Hamawi sees is that not all projects or applications will even lend themselves to automation at all points. To overcome this, he suggests that the most high value areas be automated first. Plutora has many offerings to overcome the various challenges of continuous testing, such as Plutora Test, which is a SaaS-based test management tool that can handle the entire test process.
Lambert believes that roadblocks to continuous testing fall into three buckets related to dependencies. The first is that there might be an external third-party dependency. That dependency might even have a financial fee associated with it so that every time a test is run against it, you get charged for those executions.
Another bucket is that one team can’t move forward because they are dependent on another team within their organization. That other team may be developing functionality that hasn’t been implemented yet, he explained.
Finally, teams wishing to implement continuous testing need to shift left performance testing, which might result in bottlenecks due to external dependencies, Lambert said.
Organizations can overcome these roadblocks with service virtualization. “What service virtualization allows me to do is to emulate backend system dependencies, control their functional behavior, and their performance characteristics so it can remove typical roadblocks that stop me from running my automated tests continuously,” Lambert said.
As a continuous testing company, Parasoft is able to help organizations create efficient test strategies based on their individual needs. Different techniques can be blended together to give organizations a holistic view of their environment and testing practices, Lambert explained.
“API testing becomes more important for the simple reason APIs are typically defined sooner and are also stable sooner in the cycle,” said Weishaar. “Instead of focusing on the user interface, which is usually at the end of the cycle, you can start testing sooner by using API testing.” So, incorporating API testing into the process can be another challenge.
Its founders solved some of the big problems in testing at the start, and now Tricentis is able to give its customers the tools they need to automate as much as possible and to provide a complete continuous testing platform.
Scheaffer notes that switching to a continuous testing model requires leadership and a change in mindset. “The first challenge to overcome is people,” he says.
“Another challenge many testers face is around open source software,” Scheaffer says. This is because open source software can sometimes be difficult to scale. He recommends that organizations find the right balance between open source and commercial offerings. CA Technologies has many offerings that can help organizations realize the benefits of continuous testing.
Soffer explains some of the challenges he sees with continuous testing. First of all, legacy code cannot be tested. “The closer a test is to the end-user, the flakier it is,” Soffer says.
“Unit tests are very reliable,” Soffer claims. “Integration tests are quite reliable. Automated functional tests can be very flaky.” He says that when tests are consistently flaky people will begin to ignore them, making them worse than useless.
He also notes that software that changes quickly is hard to test, even for organizations that have the best unit and integration testing practices.
Finally, “the last mile of testing is hard and expensive,” Soffer says. “As you approach full test coverage, each additional point is harder to win than the last, and may be less valuable because the tests are flakier and harder to maintain. Moreover, these tests often don’t mirror real-world conditions.”
Soffer explains that continuous testing is not usually reflective of the different devices that software would run on or the varying environmental conditions. Because of test IO’s extensive network of devices and people, it is able to bridge the gap between the lab and the real world and actually test software with real users on devices they would actually use.
“Organizations can overcome these challenges by approaching continuous testing pragmatically, with the first principle that no matter how much technology we put toward the problem of software testing, all of it is done on behalf of customers — that is, people,” says Soffer.
RELATED CONTENT: What edge do you have over other companies that provide continuous testing services?
