To make a specification like this work, a consortium such as the CISQ must not only put forth the effort to change long-established traditions in software quality, but it must also have the backing of a standards organization like the OMG to ensure an open process to foster implementation. OMG’s Soley explained how the two organizations work together toward the same goal.
“CISQ brings together, under an OMG and [Software Engineering Institute] umbrella, the best and the brightest in software artifact quality,” he said. “Then CISQ relies on the 25-year-old, open, neutral, international OMG process to ensure that open standards exist and implementations exist too. There’s no value to a standard with no implementation; OMG’s process ensures there is at least one, and usually more. This means CISQ standards get implemented and used, and are portable across toolsets and consultants. As CISQ continues to attack the metrics that matter to its member—starting with AFP, then security, then other quality metrics—we see the codebases of organizations improving.”
On a larger scale, CISQ has taken on the task of raising the structural quality of IT software. In the short term, Curtis said CISQ wants to raise awareness of the risk and cost of structurally weak software so that quality assurance addresses structural weaknesses with the same discipline and importance it addresses functional weaknesses. In the long-term, it’s about lowering the risk and cost of IT software to society, eliminating another barrier to enhancing the capabilities of computational systems.
“Measurement standards help by providing a foundation for focusing attention, setting quality targets, providing visibility, and tracking improvement progress,” Curtis said. “By using violations of good architectural and coding practice as the basis for measurement, we have seen teams learn from the results and avoid whole classes of weaknesses in future development. If the structural quality of software in business improves, we succeeded.”
Standards, standards everywhere
When it comes to specifications for standardizing the way software quality is measured, there are many ways to do it. The CISQ Software Quality Standard approaches it from a standpoint of assessing characteristics of extant code, but various specifications and standards from organizations like the ISO and IEC approach software quality from the development process, testing methods and looking at software from a product level.
Standards and specifications, such as the Capability Maturity Model Integration (CMMI), Test Maturity Model Integration (TMMI), the ISO/IEC 25000 series standards, and the ISO/IEC/IEEE 29119 software testing standard, all exist in similar, overlapping spaces. Curtis cleared up which ones compliment which, how the standards work together, and what makes the CISQ Software Quality Standard stand out.
CMMI: “CMMI is a process standard, and CMMI appraisals do not evaluate the quality of the software produced, so the CISQ standards are complementary to CMMI. In fact, CISQ measures could be used in a sophisticated appraisal to determine whether a purported high-maturity organization is producing the quality of software expected from high-maturity practices.”
OMG’s Soley added, “It was clear that while CMMI is important to understanding the software development process, it doesn’t provide feedback on the artifacts developed. Just as major manufacturers agree on specific processes with their supply chains, but also test parts as they enter the factory, software developers and acquirers should have consistent, standard metrics for software quality.”
ISO/IEC 25000 series: “The ISO/IEC 25000 series standards concern software and system product quality. ISO/IEC 25010 does a good job of defining eight quality characteristics conceptually,” said Curtis. “However, ISO/IEC 25023, the standard that defines actual software quality measures, does not do so at the internal, source-code level. Rather, it defines software quality characteristics in terms of the external behavior of the software. Thus, software quality cannot be measured and addressed until the software has been released to test or operations. The CISQ measures allow software quality to be assessed during development…thus, the CISQ measures supplement those in the ISO standard to allow a more complete life-cycle assessment of software quality.”
(Related: ISO 29119 and the software testing schism)
ISO 29119 and TMMI: “CISQ measures, based on known architectural and coding weaknesses, add a critical new dimension to the quality practices incorporated into testing process standards such as ISO/IEC/IEEE 29119 and the recently published TMMI,” said Curtis. “As was the case with CMMI, CISQ provides structural software product measures that are supplementary, since they can be used within any testing or quality assurance process that incorporates static analysis.”