Chief information officers have a number of critical mandates these days, from managing diverse IT infrastructure to helping business units develop revenue-driving services and ensuring great uptime and experiences for users. The thread stitching together all of these goals is software quality. Yet CIOs aren’t always on the same page as the quality assurance team. They may lack insight into what QA needs to deliver acceptable quality at every level — not just the core functionality. CIOs are also under pressure to have battle-proof security strategies in place, not perfect software. In this light, CIOs might be prone to scrimp on QA budgets.

The QA function has changed dramatically in recent years, in step with the wide adoption of agile methodologies and DevOps delivery practices. Faster-paced development cycles and more releases means more room for error and higher risks on both the quality and security front. CIOs don’t want to explain to the CEO why lots of customers are complaining online about the product’s clunky interface and slow load time. CIOs need to know if they are supporting QA appropriately and if their investments are paying off.

Unlike other areas of IT such as networking, QA doesn’t have a simple correlation between funding and outcomes, which makes it harder to justify spending. If you add bandwidth or switch to a better cloud service, response times will be faster, making customers happy. If you can hire more talented developers, you’ll be able to deliver more awesome products and services to the market. Investing in QA, however, is like buying insurance. You want to buy enough and the right type, so that if something bad happens, you won’t be ruined financially. If you have a well-designed QA team and strategy in place, you’ll have a much lower chance of releasing a product or update that fails and causes customer churn. If your QA team is responsible for security testing, it’s even more imperative to make sure you have purchased enough insurance to avoid a breach that causes long-term damage to the business.

While you can measure some outcomes of QA, such as test coverage and rate of production defects, these metrics don’t tell the whole story. Sometimes what you don’t see, such as a major problem that was avoided, is what’s most important.

Doing a risk assessment is an excellent way to start planning your QA budget. Here are some factors to consider in your investigation:

  • What defines risk in our company? For instance, a bank’s customer-facing application for conducting transactions cannot go offline for any significant period of time and must have rigid security protections.
  • How many people will be affected from a widespread outage or performance issue?
  • What is the likelihood of worst-case scenarios happening?
  • What is the potential market loss of application downtime?
  • What is the potential impact in the case of buggy software that annoys customers?
  • What is the complexity of the application? Complexity drives up risk and the volume of testing.
  • What is the release cycle? With frequent releases, QA will need automation tools and perhaps, more staff.
  • Are standard best practices followed for development and testing?
  • How are successful competitors investing in QA?
  • What are some industry guidelines for your application or sector regarding ratio of testers to developers?

Create the risk assessment in the same way an actuary might calculate your insurance tables. Then show the risk assessment to the CEO as a demonstration of how important it is to invest in quality and security. Make them want to buy that insurance policy.

Going through this exercise thoroughly might involve using an outside auditor or consulting firm, but you can also to speak with colleagues who are willing to share their strategies and experiences. In the end, you should have a firm grasp on software quality gaps and which pose the biggest risks to the business.

Rinse and repeat
Repeat the risk assessment at least annually. The application portfolio or user base may have changed dramatically, resulting in different requirements. Examine how defect metrics have changed, as well as trends in customer growth and defection, outages, mitigation costs from resolving problems and so on. If you set up your KPIs in advance, this will also be a good time to calculate the value of QA over the past year. You can measure your success against indicators like decreased time to market or severity of production bugs.

If QA needs suddenly increase, it doesn’t mean your budget must expand in check. Selecting the right mix of testing tools can help keep costs at bay. For instance, using script-less test automation means that your manual testers can learn how to write automated test cases with relative ease and the company doesn’t need to hire expensive automation engineers. Tools with built-in, granular reporting can help managers understand where QA practices can be improved to reduce defects and thereby the expense involved in fixing them.

There’s no such thing as full test coverage and full risk avoidance. Like any other area of IT, it’s certainly possible to overspend in QA and waste money. Yet a solid plan and process for periodically evaluating QA needs can ensure that investment is aligned appropriately with business risk.