“Both [Software Engineering Institute CEO] Paul Nielsen and I were approached to try to solve the twin problems of software builders and buyers: the need for consistent, standardized quality metrics to compare providers and measure development team quality, and to lower the cost of providing quality numbers for delivered software,” he said.
The standard lays out what measuring characteristics such as reliability, security, performance efficiency and maintainability entail. Curtis said it’s designed to develop common understanding between developers and enterprises of how software systems are analyzed, as well as how to detect architectural and coding flaws.
(Related: What kind of quality are you looking for?)
“Structural quality is measured by looking via pattern matching and related code analysis techniques for violations of good architectural and coding practice in the source code,” he said. “Code analysis technology is now capable of analyzing very large business applications whose various layers are written in different languages and involve various frameworks and technologies. For instance, modern analysis technology can automatically detect structural flaws not only within individual components, but also among components residing in several layers of the system that are part of a flawed interaction. We can detect an entry in the user interface that skips around both the user authentication functions in the next layer as well as the authorized data access routines, and accesses the data directly. This is both a security risk and a likely cause of data corruption.”
Using code analysis tools in accordance with the CISQ Software Quality Standard, Curtis explained how executives, application managers and developers could use these measures to identify which of the applications present the greatest risk to their business or involve the highest cost of ownership.
“Application managers can use these measures to track trends across releases in the structural quality characteristics of their systems,” Curtis said. “Developers will be able to discover multi-component, inter-layer weaknesses that are impossible to detect until they are integrated in a build, weaknesses that must take priority because of breadth of their potential damage. Developers can use the analysis output to identify and prioritize the riskiest or costliest flaws for correction. Finally, customers can use these measures to evaluate the quality of the software their system integrators and outsourcers are delivering.”
As for what to do once these software quality risks are uncovered, Curtis said that the minimum requirement for IT organizations is to set quality measurement thresholds for critical business systems, along with monitoring compliance with thresholds upon each delivery or software release. He also recommended removing unacceptable flaws in future editions.
Yet in order to manage application development and maintenance quantitatively (the way management uses numbers to run the rest of the business), Curtis said IT professionals should embrace the standard.
“We have entered the era of nine-digit defects—software flaws that cost the company over $100 million,” Curtis said. “Knight Trading, RBS, Target, to name just a few, have suffered extensive financial and reputational damage from software disasters. Structural flaws in business-critical systems and the level of risk to which they can expose the business have become boardroom issues. Executive management needs a way to evaluate software risk and cost, and they do not want proprietary measures. They want standards that help them govern and perform due diligence based on sound measures developed from industry experience that assess risk and cost.”
Adoption and software quality’s future
CISQ’s members and sponsor companies, including IT executives, system integrators and software technology vendors, are still working to improve the standard and are debating which software metrics and code quality weaknesses the standard should include, supplemented by the OMG’s specification revision process. At the moment, the OMG is still in the process of approving some aspects of the standard, but Curtis believed adoption of the standard will see a sharp rise.
“The CISQ Quality Characteristic Standards are still going through their approval process at OMG, so we are ahead of the adoption phase,” he said. “However, if the rapid adoption of CISQ’s standard for Automated Function Points is any indication, we expect many IT organizations will begin using the quality characteristic standards as SLAs in their outsourcing agreements very soon after the standards are approved by OMG. The standards were created because IT customers and suppliers both asked for them and participated in their development. Therefore, adoption will be driven by a pull rather than by a push.”