Organizations are moving to continuous testing (CT) out of necessity because business competitiveness demands faster release cycles. In fact, teams can’t deliver on the promises of DevOps and CI/CD if testing isn’t part of continuous processes and the pipeline.

Forrester Research VP and principal analyst Diego Lo Giudice and some of his colleagues, recently published a report that includes 12 essential must-dos that span people, practices and technology. The following is based on a recent interview with Lo Giudice in which he shared insights that are explained in greater detail in the report.

People
Continuous testing requires testing team transformation. Instead of having a centralized test center where all the testers reside, executing and managing all the tests, there’s now a hub-and-spoke structure which includes a small center of excellence and testers that are assigned to different teams.

RELATED CONTENT: Continuous testing isn’t optional anymore

“The traditional way, you had a development team that would write the code and throw it over to the test center to do the testing to find bugs. That’s not the way we operate today because testers are in the Agile teams and what’s in the central team is a small team that’s focusing on best practices,” said Lo Giudice. “The central team is maybe recommending tools, harvesting the good practices from different teams and formalizing and sharing them among the teams. So, there’s a shift from a centralized test center to a federated test center.

The testers working in Agile teams need Agile skills including the ability to talk with developers and product owners from the business.

“That’s a different testing persona,” said Lo Giudice. “The testing persona of the past would look for bugs and be happy he found a lot of bugs. Now [that developers and testers are] on the same team, they have shared goals. Quality is one of them. The tester helps prevent bugs from happening so the tester gets involved earlier on in designing the test cases, helping the developers formalize the unit testing, making sure that developers are doing their unit testing and that they’re covering as much code as possible. [The testers are] helping developers build better quality code from the beginning.”

Also, to align their efforts and jointly produce better quality code, developers and testers also need to share common metrics.

“In the past, we never measured if the level of automation is improving. We never measured how long automation takes because when these teams measure the execution of automation, they check in code in a CI tool and execution kicks off. If it’s suddenly taking longer, then something is going on,” said Lo Giudice. “That’s an indication that the release will be stopped, that the code that was checked in will go back to the team to figure out what the problem was.”

Practices
Behavior-driven development (BDD) is one of the practices teams are adopting. Many of them are using Cucumber, a BDD development tool and Gherkin, its ordinary language parser because when test cases and test scenarios are written in ordinary language, everyone can understand them.

“It helps the collaboration between the product owner from the business, the tester and the developers. The product owner will write what he wants in terms of the behavior of the application together with the test cases and then people will understand that language. He can start thinking about how to write the automation for it and depending on the tools that might be generated from the DSL,” said Lo Giudice.

Other teams have adopted test-driven development (TDD), which differs from BDD.

“TDD is different because it impacts the life cycle. It’s writing the test cases and then the code that passes the test cases,” said Lo Giudice.

Shifting left is another popular practice that involves testing as soon as a new sprint or product development starts. More types of testing have been shifting left over time, and that will continue to be the case. Right now, a lot of organizations are focused on shifting performance testing left because leaving it to the end is too late.

“Testers are part of the team and involved early on. It’s about starting testing, and unit testing is one way of shifting left, but it’s about the testers working with the product owners and the team defining test cases or user acceptance criteria right away when we start writing the user stories in the background,” said Lo Giudice.

Service virtualization is also essential for shifting testing left because developers and testers can mock up resources instead of filing a ticket and then waiting for operations to make a resource available or competing with others to access a resource.

Forrester stopped covering service virtualization separately because it doesn’t have its own market, so it’s now included as part of continuous testing.

“You don’t need the full service virtualization capabilities that the tools three to five years ago were offering, but simplified versions that help you do a stub very quickly,” said Lo Giudice.

Teams also need to shift testing right as well as left.

“It’s monitoring the view into production. If you’re deploying your features frequently in production and the developer can monitor some of the code that they’re deploying, they can prevent performance issues from happening,” said Lo Giudice.

Finally, exploratory testing is replacing the old way of manual testing. Manual testing isn’t going away, but its uses are diminishing.

Technology
The tech stack is more focused on smart automation than traditional test automation. Smart automation uses AI and machine learning to help developers focus on what matters, which simplifies and speeds testing.

“Smart automation tools leverage machine learning and AI to generate from requirements more precise test cases that would optimize the business coverage, so that’s at the design level,” said Lo Giudice. “But there’s also automation of test execution. When code gets checked in, do I have to run all my regression tests or based on the change can I figure out the ones that need to be run and shorten the execution?”

API testing is also important because developers are writing more API and microservice-based applications. Beyond that, there should be fully layered testing and version control with all assets stored centrally.

“All the testing assets necessary to speed up the automation end up being stored together with the code, so you store the code that’s being tested, the code that we use for writing the test cases, and all other assets so I can version all that,” said Lo Giudice. “If I find a bug and need to review and update the test automation, I can do that very quickly, so in the technology stack, CI/CD integration with application life-cycle management remains fundamental.”

For advanced performance testing, test data management is recommended.

“You can’t use the old way of doing test data generation when we’re cycling fast on testing continuously,” said Lo Giudice. “You have to have something that integrates into the sprint or the life cycle and updates the data all the way through.”

Self-service provisioning of test environments is also essential. That’s accomplished in the cloud, spinning up and spinning down test environments.

Expect AI and machine learning to impact vendor rankings
At the time of this writing, Forrester is about to release its Forrester Wave on Continuous Test Automation. Of the 26 criteria used in the wave, more than half of criteria (15 or 16) focus on functionality.

“A very large portion of those had a question around how and why are you using any ML or AI in this capability,” said Lo Giudice. “The response was the vendors have finally moved onto this, so this year you’re already seeing the use of AI in the tools and the way they’re using it. They’re using it to make things smarter.”

How, exactly, will be covered in the August issue of SD Times, which will include a deep dive on machine learning and AI in the context of CT.