Continuous testing in a fast-paced agile and DevOps world

In a world where businesses not only have to become software companies to compete, but deliver higher quality software to market faster, what does this mean for testing? Businesses and software teams have become so concerned with developing and delivering in an agile fashion, they have had to stop and rethink their entire testing process.

This is because traditional or legacy testing tools and processes don’t comply with speed, according to Aruna Ravichandran, vice president of DevOps product and solutions marketing at CA Technologies. To compete in a world where high-quality app experiences are the expectation, testing needs to be efficient and effective, and that requires a continuous testing approach, according to Ravichandran.  

“Testing needs to be as fast and as dynamic as our agile development processes, otherwise we are going to have delays in developing and shipping it to operations,” said Wolfgang Platz, founder and chief product officer of Tricentis.

Platz explained agile has always been thought of as a development initiative from the beginning, which was too narrow. While it accelerated the speed of development and brought value to customers faster, agile development needed to take operations into consideration. “All of a sudden we weren’t talking about agile anymore, but talking about DevOps,” said Platz. But as soon as people started to pursue a DevOps agenda, they realized there was an ugly stepchild in the middle that needed to be taken care of, which is testing.

“In many ways, testing is the final frontier of DevOps and continuous delivery. To release the high-quality code that enables businesses to stay ahead of their competition faster, organizations must adopt next-generation testing practices that enable them to test early, often, automatically and continuously,” said Ravichandran.

However, according to Marc Brown, product portfolio strategist for Parasoft, continuous testing should actually be a prerequisite for continuous delivery and continuous integration, and continuous delivery should be a prerequisite for people who want to get to DevOps. “You have to think of it as a set of building blocks where at the core you have organizations that want to have more flexibility and have adopted agile, they then have to adopt continuous testing to ensure what they are developing can be tested on an ongoing basis,” he said

But this fast-paced way of working creates many challenges for testers, according to Jason Hammon, director of product management at TechExcel. Hammon explained testers now have to struggle with the lack of time and need to respond as quickly as plans change. In an agile world, testers no longer have the luxury of a clear roadmap on what they are going to test, time to prepare the tests, and then time to execute on the testing.

But if teams are able to implement a good continuous testing strategy with the proper tools to support that strategy and scale, it will lessen the pain, according to Rod Cope, chief technology officer at Rogue Wave. With continuous testing, Cope says teams get the benefit of always having code in a good state. They no longer have to write code, test it, and then wait to find issues, stop the presses and go back and fix things. Instead, they are able to constantly take pieces of the solution and constantly test, instead of waiting for a big-bang delivery and testing phase.

“You can get much more done much more quickly with better quality, better security and less rework so you are getting more bang for your buck out of your development team,” said Cope. “You clearly can’t take a year anymore to get a product out the door; you have to be much more agile, and continuous testing is absolutely mandatory in today’s world. You really can’t afford to have problems in the field. You have to get it right the first time, and you have to do it very quickly.”

The resurgence of testing
For so long testing was an afterthought, but today the mantra ‘test early and test often’ is drilled into the heads of developers and testers. This is largely due to the increased speed of delivery and the push to adopt agile and DevOps, according to HPE’s Emo.

Even the smallest releases can have a large impact on the end user experience, and if an application delivery isn’t what it was expected to be, that response can spread around the market very fast, Emo explained. It is now more important than ever to put testing techniques in place throughout the entire life cycle so testing does not become a bottleneck, and teams can keep up with continuous quality.

The acceleration of development is a correlation of the consumer expectation. People expect the first version of something to work very well and be very obvious to use. They want their solutions to be simple to understand, but do way more things, according to Mobile Labs’ McFall. That demand for quality is driving the demand for testing. “You have to be developing all the time to keep up with the timeline demands, and the only way to meet the quality demands is to be testing it constantly,” he said.

McFall also explains the concept that software is eating the world is also having a huge impact on testing. Today, everything has to become software in some way, shape or form. The digital transformation is adding more complexity to the testing world because devices are starting to take many different forms, provide more capabilities, and interconnect between features and functions.

This software complexity is not only turning up the dial on speed, but also changing the ways to deliver. Because of this, more teams are starting to move to a software architecture that is extremely agile such as service-oriented architecture or microservice architecture, according to Emo. “This is great from an  architectural and developer agility perspective, but it gives you a lot more to manage, and a lot more dependencies,” she said. In addition, the software is being deployed across a multitude of operating systems, mobile environments, mobile devices, and Internet of Things devices, Emo added.

To deal with this, teams need to be continuously testing. McFall explains it doesn’t always have to be test everything all the time, but something should always be being tested, whether it is unit tests, interoperability, or white box/gray box testing. “Teams need to be able to anticipate problems, and that means feedback loops need to be automated and process loops need to be automated to be able to keep up,” he sad. 

RELATED CONTENT: Vendors make a case for their continuous testing solution

Automation: The key to continuous testing
One way continuous testing can help pick up the pace of testing is with automation. Agile and continuous testing have a tremendous amount of potential value to add to organizations, but if they aren’t fully automated it can be very taxing on the organization and create a lot of overhead, according to Parasoft’s Brown.  

Tricentis’ Platz explains automation allows teams to test at the speed of agile and DevOps; reduce the time and effort it takes to obtain realistic and accurate test results; minimize test maintenance; and reduce false positives.

In order to successfully deploy test automation, teams need to have a good understanding of what test automation is, how it can be leveraged, and where it is going to be most successful based on the type of product, according to TechExcel’s Hammon.

This requires connecting a number of tools together within the overall software development chain, Brown explained. “Automation tools need to be very easily integrated with the continuous testing tools to really power the continuous testing process,” he said. Brown adds organizations need to adopt tools that enable service virtualization to support automation. “It is the easiest way to create test environments on demand without the effort. The process and vision of continuous testing needs to be completely coupled with the technologies and automation to realize the value, otherwise it is going to create a lot of overhead for teams,” Brown added.

In addition, test teams should be comfortable with developer assets to write test cases in the form of automated scripts, according to Kelly Emo, director of life-cycle and quality product marketing at Hewlett Packard Enterprise (HPE). “They need to be comfortable with IDEs, programming languages and application models.  Whether Java, C#, Java script or other, they have to be comfortable working at the code level to create the most effective and reusable automation assets,” she said.

However, businesses should note that continuous testing is not just another buzzword for test automation. According to Dan McFall, president of Mobile Labs, automation is just a piece of the continuous testing puzzle, and it doesn’t mean that every test is going to be automated or even continuously tested. “You have to design the process and figure out where things are going to go, and then you figure out which technologies are going to come fit in and be in support of that,” he said.

In addition, automation doesn’t just apply to testing. Automation can be applied to data transfer, data flows, deployment, builds, and build management, according to McFall.

According to Rogue Wave’s Cope, in an ideal world everything would be fully automated, but realistically that is difficult unless you invest early on in the codebase, design and architecture. Cope adds at least on the UI side teams are always going to have to have some manual check and review in place if not heavy testing with every release. This is because things as simple as fonts and colors are hard to automatically test.

In addition, it requires a complex infrastructure to set up application servers, web servers, databases and configure all the different moving parts to be deployed in an isolated environment and run all the tests. However, more teams have been moving to containerization to make it easier to get environments up and running quickly, automate the process, and automate all the testing around it, Cope explained.

Other pieces of the puzzle
On top of automation, there are a multitude of other key strategies and tools necessary for a continuous testing workflow. “The continuous testing journey is a change. You can’t just jump into things, you need to take care of several other things first,” said Tricentis’ Platz.

For instance, Tricentis provides a continuous testing maturity model where customers can assess where they are today, and come up with a plan on how to get to the most mature stage.

“While we’ve all heard about organizations that were ‘built to last,’ the truth is, the only way an organization can survive and thrive today is to make sure they are ‘built to change,’” said CA’s Ravichandran.

Parasoft’s Brown says organizations need to invest in their developers, testers and release managers, and train them on what it means to be continuously testing. Then, you must train them on the environment and on the tools so they realize the value of what they invest in versus buying a product, not seeing its full potential, and becoming shelfware. “Training is probably the biggest element outside of the necessary tools and technologies needed to really power this, and make sure customers are successful at their continuous testing endeavors,” he said.

That requires a lot of communication, according to TechExcel’s Hammon. Teams need to understand why they are moving in a certain direction and choosing specific tools.

Communication is also a good skill QA engineers should have. That way, if an issue comes up, they can easily explain to the developer what happened and why it needs to be fixed. “They could also communicate with people that may be on the marketing or product management side to fully understand the initial goal of the feature to make sure they are really able to provide test cases that verify and validate the feature is actually implemented correctly,” Hammon said.

In addition, communication means taking into account customer feedback so teams can integrate that into the functionality of their system and make sure they are testing the right things. Having a good feedback loop in place gives teams the ability to see the overall health of the solution and provide early warning into issues, according to Mobile Labs’ McFall. “Without metrics or a good strategy, teams will be lost on whether or not they are actually delivering what the customer is expecting. If you don’t have a clear destination, just because you are going fast doesn’t mean you are going to the right place.”

HPE’s Emo explains there will always be some defects that leak out into production, but if you have that closed feedback loop and communication and an incident is found in production, it can get back to the agile team fast and they can quickly fix the problem.

CA’s Ravichandran adds to have a successful continuous testing strategy, teams need integrated solutions that enable “1) test environment simulation, 2) automatic test case creation, even from requirements, 3) on demand test data management, 4) always available SaaS-based performance testing, 5) open source integrations with tools like JMeter, Jenkins, Selenium, Appium, and 6) automated application performance monitoring and post-production performance testing to achieve continuous feedback.”

Who is responsible for testing?
Agile development requires cross-platform teams. This means that not only are teams made up of QA engineers, developers, and project managers, but the responsibilities are shared amongst the team members. Because of this, CA’s Ravichandran says testing and quality is everyone’s job.

“Continuous testing makes testing activities part of everyone’s role…from the business analysts, to the developers, to the performance engineer, to the operations manager… all are responsible to make sure the end users have a fantastic experience with their software,” Ravichandran said.

While this may be true, TechExcel’s Hammon believes QA and testing teams should still lead the charge. “It is great to have developers do unit tests on what they are checking in, but you still want to make sure the overall work of the individual developer when put together with the rest of the solution isn’t resulting in additional problems. You still need someone to make sure the overall product is what it was initially intended to be,” he said.

According to HPE’s Emo, testers have a mindset and specialty for solving functional, regression, performance and security issues that developers don’t have. But the testers should make sure they are still a part of the agile team, and participate side by side with development. This requires new skills such as the ability to understand the fundamentals of software development. “Understand agile processes and be able to understand good software design patterns and architecture. This way [the QA team] can focus on how to test in a way that will surface issues related to the fundamental way the application is constructed. Performance engineers should have strong skills in the performance characteristics of different application types, architectures and patterns. [And] security engineers should have strong skills on application layer security vulnerabilities,” she said.

Some businesses come up with a test Center of Excellence (CoE). Traditionally CoEs provide a large test team that delivers test as a service to the development team, but this provides distinct disadvantages, according to Tricentis’ Platz. Platz explains in a traditional CoE, communication isn’t as connected as it should be, causing testers to be informed late about changes.

CA’s Ravichandran recommends that CoEs become Centers of Enablement. This allows teams to collaborate in an environment where developers are integrated into testing from the beginning. “When testing is conducted earlier, the entire team is focused on quality, and a culture of “getting code done right the first time” saves time and reduces the number of iterations teams need to perform for a particular code,” Ravichandran said.

Get your guide to continuous testing tools here.