“CDT is better than the Factory approach because there is no evidence that testing can be turned into a rote procedure,” he said. “The very nature of testing is creative and critical thinking. It is a form of design work. It is inherently exploratory. Anyone who thinks they can make this into a factory-like process must offer some theory that justifies this, and some example of how that theory has ever led to success in practice.
“No one does this. Factory school testing is very popular, not because it works, but because it seems easy to manage… We in CDT pride ourselves on being able to explain what we do and why we do it. There is nothing mystical—it’s just a matter of skill. That skill can be systematically taught and learned.”
What about automation?
One glaring aspect of modern software testing only tangentially acknowledged by either side of the schism is automation. The standard defines a few concepts of automation and covers one narrow aspect in regards to keyword-driven testing, and the context-driven testers behind the petition are “concerned about the over-automation of testing,” according to Bach.
“We avoid the phrase “test automation” because testing cannot be automated,” Bach said. “We call that “tool-supported testing” or “automation in testing.” We make extensive use of tools…We think of automation the way skilled carpenters think of automation: we love our power tools, and we can also operate without them. What we are against is the de-skilling of the craft through the process of attempting to automate things that actually can’t be automated.”
In the current testing climate (where the advent of agile and added regression risk of constantly perturbing codebases have increased the need for automation), leaving automated testing out of the argument severely damages its credibility, especially given the proliferation of countless high-quality open-source and commercial tools.
Keyword-driven testing is a test automation framework concerning automated system testing and GUI testing, but that alone does not make the standard a representative guide to automation.
Black believed KWT is an important component of the solution, but only a small part. “When you think of these Continuous Integration frameworks like Hudson and Jenkins, and all the tools you can plug in for static analysis, automated unit testing, test-driven development and behavior-driven development, automation is enormous,” he said.
“It’s not the one solution, but it’s certainly part of it. It’s not either this or that; it’s all of the above. We have to come at a testing problem a lot of different ways, and automation is an effective test strategy especially against regression risks.”