“We can automate functional testing at multiple levels to some extent, though there are some validation tests which may not be so easy to automate,” said Black. “At some point, we’ll have AIs that can do ethical hacker kinds of tests, but we don’t have that now, so if you want to do a penetration test, you need a human ethical hacker to do that.”

Localization testing tools are using machine learning to test translations. They’re not perfect yet, nor is Google Translate which is available via an API, so a human is still needed in the loop. Humans also need to be involved in accessibility testing and portability testing to some extent.

“You need to start with what do we need to test and if you let the test automation determine what your test coverage is going to be, that’s likely to make you sad,” said Black. “I made that mistake when I was an inexperienced test manager and it’s not a good thing. Identify all the things you need to cover and then figure out the best way to cover those things.”

Drive higher value with a test strategy
The inclusion of analytics, AI, and machine learning in tools is arguably “cool,” but the outcomes they help enable and the effectiveness of testing generally can be improved by having an overarching test strategy.

“This is a very big problem. Software engineering is notoriously fad-driven and hype-driven,” said RBCS’s Black. “I’ve worked with a number of clients who were heavily into automation and if you look at what they were doing, there was no strategy. I ask questions that are pretty obvious like, ‘What are the objectives you’re trying to accomplish? For each one of those objectives, show me how you’re measuring testing and efficiency, show me the business case for your automation. What’s your ROI? Without a business case, it’s not a tool, it’s a toy.”

There is a temptation to go to tools first and think later, when the reverse yields better results. Starting with tools results in a test strategy by default rather than test strategy by design.

“I think many people would say [that having a testing strategy] is an old-school approach, but it’s the right approach because if you’re just going with tool A or tool B, you’re trusting your approach to that tool and that tool may not provide what you need,” said Voke’s Lanowitz. “I think you need to take a step back and decide what you’re going to do.”

Ultimately, the testing strategy and execution need to tie together in a way that aligns with business objectives, but more fundamentally organizations have to cultivate a culture of quality in the first place if they want their CT efforts to be stable.

“Understand that nothing we’re doing can be 100%. What we’re really doing is trying to minimize our risk as much as possible and mitigate for things we weren’t expecting,” said RBCS’s Black. “Continuous delivery is a process that lives forever. As long as the product is alive, we’re testing it.