Testing has always required tools to be effective. However, the tools continue to evolve with many of them becoming faster, more intelligent and easier to use. Continuous testing (CT) necessarily recognizes the importance of testing throughout the software delivery life cycle. However, given the rapid pace of CT, tests need to run in parallel with the help of automation. However, that does not mean that all tests must be automated.
The nature of “tools” is evolving in the modern contexts of data analytics, AI and machine learning. Nearly all types of tools, testing or otherwise, include analytics now. However, tool-specific analytics only provide a narrow form of value when used in a siloed fashion. CT is part of an end-to-end continuous process and so success metrics need to be viewed in that context as well.
AI and machine learning are the latest wave of tool capabilities which some vendors overstate. Those capabilities are seen as enablers of predictive defect prevention, better code coverage and test coverage, more effective testing strategies and more effective testing strategy execution. It takes some basic knowledge of AI and machine learning to separate tools that actually include those features versus other tools that only sound as if they provide those features.
Diego Lo Giudice, VP and principal analyst at www.forrester.com Forrester Research outlined some of the testing tools needed for a CT effort have been included below. The list is merely representative as opposed to exhaustive:
- Planning – JIRA
- Version Control – GitHub
- CI – Jenkins
- Unit testing – JUnit, Microsoft unit test framework, NUnit, Parasoft C/C++test,
- Functional testing –Micro Focus UFT, TestCraft
- API testing –Parasoft SOAtest, SoapUI
- UI testing – Applitools, Ranorex Studio
- Test suites – Smart Bear Zephyr, Telerik Test Studio
- Automated testing (including automated test suites) – Appvance, IBM Rational Functional Tester, LEAPWORK, Sauce Labs, Selenium, SmartBear TestComplete, SOASTA TouchTest, Micro Focus Borland Silk Test
- CT – Tricentis Tosca
Analytics and AI can help
Test metrics isn’t a new concept, although with the data analytics capabilities modern tools include, there’s a lot more than can be measured and optimized beyond code coverage.
“You need to understand your code coverage as well as your test coverage. You need to understand what percentage of your APIs are actually tested and whether they’re completely tested or not because those APIs are being used in other applications as well,” said Theresa Lanowitz, founder and head analyst at market research firm www.vokeinc.com Voke. “The confidence level is important.”
Rex Blackpresident of testing training and consulting firm RBCS, said some of his clients have been perplexed by results that should indicate success when code quality still isn’t what it should be.