Automation is just like every other technology in that it takes effort to be successful. It’s another strategy that can be incorporated into the software testing life cycle, but issues come from testers figuring out how they can build it into a process and determining what solutions or tools are right for their company.
Over the past few years, testing went from 100% manual to incorporating automation so tests could be fast and repeatable. With mobile, agile and DevOps as driving forces, the payoff and ROI for automation that was promised years ago is finally happening, according to Mobile Labs vice president of mobility solutions Dan McFall.
Since businesses are moving to faster releases, the time to market has changed. According to Anand Kamat, group program manager at Microsoft, getting a software idea to become reality in a short timeframe puts pressure on organizations.
Although the focus is on automation (and rightfully so), manual testing is still relevant and takes many different forms in agile, said Kamat. Developers are continuing to incorporate more automated tests, with “test early, test often” still the guiding principle.
The risks of automated testing
Automation isn’t a one-size-fits-all solution. Companies have to complement test automation metrics with data from manual testing, exploratory testing, user acceptance testing, and testing in real-world customer environments to form a comprehensive view about quality of the product, according to Anand Kamat, group program manager at Microsoft. He suggests considering the following risks before getting started:
- Most often automation does not cover 100% of use cases, but having a consistent pass rate of 100% can give you a sense of “false confidence” that is fatal.
- Code coverage is not a reliable metric for ensuring end-to-end quality, but is often used as a measure to gauge the effectiveness of test automation.
- For connected applications involving multiple components, your “one-box” setup is not a real-world scenario. If you are not testing in an “integration environment,” you are not testing with right dependencies.
- With frequent changes in user experience, ROI on UI Automation might be limited for multi-channel applications. Having a services/API testing strategy combined with exploratory testing might be a better alternative.
Speed over quality
Because of the rise of automated tests, the need for speed continues to stay on companies’ minds. Matt Brayley-Berger, worldwide product marketing manager for HPE, said that his company always likes to ask the question, “Would you like to have better quality or faster speed?” They pondered the question, eventually reaching a consensus that organizations are finding ways to remove a lot of barriers that would have already lowered quality, according to him. This means they can still focus on the speed of release without sacrificing quality. And, if testers became more technical and had to work with development a little more closely, it forces the DevOps conversation sooner.
“It’s not the solution, but maybe that behavior is creating an environment to have more productive conversations with evolving testing and evolving quality,” said Brayley-Berger.
The shortcuts that take place during the development process can take a one-month turnaround to a one-week timeframe by simply reducing the amount of testing that is being done. According to Walter Capitani, a product manager at Rogue Wave, this is what companies are doing. He said they take shortcuts in testing by reducing the tests or by doing things after they’ve finished the release because they figure if they find a quality problem, they can always “patch it later.”