“This means that consumers of software testing services and testers themselves had no single source of information on good testing practice. Given these conflicts and gaps, developing an integrated set of international software testing standards that provide far wider coverage of the testing discipline provided a pragmatic solution to help organizations and testers.”
In recounting the road to a complete standard, Reid explained that because of the evolving nature of the testing industry, many parts of the standard such as Test Processes were devised from scratch to ensure that factors like agile life cycles and exploratory testing were considered. The Keyword-Driven Testing standard only came into existence in 2012, and a working draft of an as-yet unapproved standard on Work Product Reviews is still without a timeline for publishing.
The working group has a long list of co-editors and members, consisting of academics and testers from universities, startups, companies large and small, and even governments. Two dozen countries are represented as well.
“The question is who recognizes this standard and who created the standard,” said Gartner’s Murphy. “Often many of these standards are also looked at as ivory tower and process-centered, not reality-based or agile-driven.”
In reference to the WG26 co-editors, Murphy said, “You do get a couple consultants and academics. Also since Mindtree [general manager Sylvia Veeraraghavan] was an author, it will be interesting to see if acceptance comes from the larger testing service providers.”
Still, Reid stressed the working group’s focus on the openness and comprehensiveness of the standardization process. He detailed a “mirror panel”—similar to the concept of an electoral college—where WG26 members represent and consult with their home countries, testers from a specific industry, specific roles such as testing consultants, or testing interest groups to review and comment on drafts of each standard, all in the name of “consensus.”
“Members of [WG26] are well-versed in the definition of consensus,” Reid said. “The six years spent in gaining consensus on the published testing standards provided us all with plenty of experience in the discussion, negotiation and resolution of technical disagreements.
“However, we can only gain consensus when those with substantial objections raise them via the ISO processes. The petition talks of sustained opposition. A petition initiated a year after the publication of the first three standards, after over six years’ development, represents input to the standards after the fact and inputs can now only be included in future maintenance versions of the standards as they evolve. It is unclear if ‘significant disagreement’ reflects the number of professional testers who don’t like these standards or the level of dissatisfaction with their content.”
The term “consensus” can be a hazy one, though. Rex Black, a software tester and president of test consulting, outsourcing and training firm RBCS, believed that Reid genuinely wants to make a difference in testing, but that the representativeness of the standard is ultimately beholden to a small committee.
“It represents a consensus of the people who participated in the standards committee, just like any standard,” Black said. “That’s always the case, and it’s one of the limitations of any standard or pseudo-standard. You just have to hope those people are reputable and did their homework to make sure they understood what other perspectives are out there.”
Testing is a fragmented community to begin with. Various schools of thought compete with each other in the marketplace of ideas, with no consensus in the field even among “recognized experts” about the definition of testing. In that light, trying to apply a standard to the craft only serves to split the testing world down the middle.
One of the loudest voices heaping criticism on ISO 29119 is that of software tester, author and speaker James Bach, founder and CEO of test consulting company Satisfice and a charter member of the ISST. He is also a proponent of the context-driven testing philosophy on which the ISST is founded.