Today’s DevOps and “Continuous Everything” initiatives require the ability to assess the risks associated with a release candidate—instantly and continuously. Yet, as the release date looms, development teams are still focused on answering the question, “Are we done testing?”
Fundamentally, this is the wrong question. It ties the concept of “quality” to static tests that produce multiple, independent and primarily binary data points of pass or fail. This approach results in a lot of data points, but not the information needed to help the business understand the real impact to the end-user experience.
(Related: Putting the test back in DevOps)
Understanding the specific risks associated with each release candidate becomes mission critical as organizations attempt to accelerate the release cycle. Without this visibility and knowledge of the impacts to the business, managers are unable to make the appropriate tradeoff or timing decisions for releasing software.
Instead of “Are we done testing?” we should be asking, “Does the release candidate have an acceptable level of business risk?” This new question is much more complex than it seems at the surface. It carries a few critical assumptions:
- The inherent business risks associated with a given application and the particular release candidate are well defined.
- There is an understanding of how to measure each of these defined business risks.
- A baseline and thresholds are established for defining what constitutes an acceptable level of risk. Some business risks might have zero tolerance and no thresholds for acceptance.
- Automation is in place to continuously assess the state of the application versus these defined risks.
This is why the concept of Continuous Testing is so critical. Continuous Testing provides an automated, unobtrusive way to obtain immediate feedback on the business risks associated with a software release candidate. It balances the traditional bottom-up tasks associated with software development and testing with a top-down approach focused on safeguarding the integrity of the user experience while protecting the business from the potential impacts of application shortcomings. Given the business expectations at each stage of the SDLC, Continuous Testing delivers a quantitative assessment of risk as well as actionable tasks that help mitigate risks before they progress to the next stage of the SDLC. The goal is to eliminate meaningless activities and produce value-added tasks that drive the development organization towards a successful release.
Continuous Testing is not simply more test automation… nor is it a “plug-and-play” solution. As with all process-driven initiatives, it requires the evolution of people, process and technology. We must accommodate the creative nature of software development as a discipline, yet we must face the overwhelming fact that software permeates every aspect of the business—and software failure now presents the single greatest risk to the organization.
Continuous Testing (when executed correctly) provides four major business benefits. First, it results in clearly delineated business risks associated with each application in the organization’s portfolio, including measurement standards for assessing the level of risk. It guides business and technical teams to collaboratively close the gap between business risk and development activities.