“One thing that happens is sometimes [there is] a unidimensional view of coverage, and I’ve seen this with clients where they say, ‘How could we possibly be missing defects when we’re achieving 100% statement coverage?’ and I have to explain you’re testing the code that’s there, but how about the code that should be there and isn’t?” said Black.
Similarly, there may be a big screen dashboard indicating that all automated tests have passed, so the assumption is all is good, but the dashboard is only reporting on the tests that have run, not what should be covered but hasn’t been covered.
Forrester analyst DIego Lo Giudice referenced a “continuous testing mobius loop” in which data is generated at each step that can be used to do “smarter things” like deciding what to test next.
“If you start using AI or machine learning, you can start to predict where the software may be more buggy or less buggy, or you can predict bugs and prevent them rather than finding the bugs and validating or verifying them,” said Lo Giudice. “On the other hand, it’s part of a broader picture that we call ‘value stream management’ where each type of stakeholder in the whole end-to-end delivery process has a view.”
Quality is one of the views. Other views enable a product owner to understand the costs of achieving a quality level, what has been released into production, how long it took and that the value it’s generating.
Sean Kenefick, VP, analyst at www.gartner.com Gartner said he’s currently working on a project that involves looking at AI and its relationship to software testing. Importantly, there are two ways to view that topic. One is testing systems that incorporate AI. The other is using AI as part of the testing effort.
“I think AIs are going to have quite a significant impact on automated testing because they’re going to allow us to solve some really thorny problems,” said Kenefick. “Some of my clients are in the video game space where, unlike an insurance package or a banking package that have right answers determined by banking regulations or GAAP, games are disconnected from the real world.”
For example, in a game, players may have wings that enable them to fly. Moreover, they may be able to fly at a speed that would dismember a physical body in the real world. Traditional automation tools have done a poor job of bridging the gap between fictional scenarios and the real-life expectations of humans, such as hair flapping in the wind. AI enhancement could help.
Automated testing is ripe for AI enhancement from a number of perspectives such as identifying tests that should be automated and suggesting new types of tests, but again, even with AI it may not be possible or even wise to automate all types of tests.