Is automated testing worth the expense?

Real test practitioners will show how test automation solved many of their quality issues when the Automated Software Testing and Quality one-day virtual event returns on May 16. Produced by software testing company Parasoft, among the topics to be discussed are metrics, how automation can significantly cut test time, shifting testing left, the use (or not) of generative AI, the synergy between automation and service virtualization, and more.

“We’ve worked really hard to make sure that most of the sessions are coming from the practitioner community,” said Arthur Hicken, chief evangelist at Parasoft. “So people are telling you how they solved their problem – what metrics they use to solve the problems, what the main challenge was, what kind of results they saw, you know what pitfalls they’ve hit.”

As for AI in testing, Hicken said Parasoft has created AI augmentations at every aspect of the testing pyramid, which he acknowledged is getting “kind of long in the tooth,” before adding that it still is useful, helpful advice. “Whether it’s static analysis, unit test, API testing, functional testing, performance testing, UX testing, we’ll talk about how these different things will help you in your day-to-day job.”

He went on to say that he doesn’t believe the things he’s talking about are job killers. “I think they’re just ways to help. I haven’t met any software engineer that says, I don’t have enough to do, I’ve got to pad my work with something. I think just being able to get their job done will make their life better.”

On the subject of generative AI, Hicken says it can be quite smart about some things but struggles with others. So, the more clearly you can draw the boundaries of what you expect it to be able to do, and the more narrow you can scope it down, AI just does a better job.

This, he said, is true of testing in general. “Service virtualization helps you decouple from real-world things that you can’t really control or can’t afford to play with,” he said. “Most people don’t have a spare mainframe. Some people interact with real-world objects. We see that in the healthcare space, where faxes are part of a normal workflow. And so testing becomes very, very difficult.”

Further, he said, “As we use AI to start to increase the amount of testing, we’re doing the permutations, we run into a data problem, we just don’t have enough real data. So it starts synthesizing virtual data. So the service virtualization is a way to synthesize data to get broader coverage. And because of that, there’s always a temptation to use real-world data as your starting point. But in many jurisdictions, real-world data is a pretty big no-no. GDPR doesn’t allow it.”

So, in the end, the question remains: How do you know it was worth it? What did you do to measure? Hicken said, “I don’t believe there’s a universal quality measure or ROI measure; I believe there are lots of fascinating different things that you can look at that might be interesting for you. So I would say look for that.”

Hicken also noted, humorously, that if test automation did not deliver value, the speakers he sought out for ASTQ would not have returned his calls. 

There is still time to register to learn more about automated software testing and Parasoft.