Today’s DevOps and “Continuous Everything” initiatives require the ability to assess the risks associated with a release candidate—instantly and continuously. Yet, as the release date looms, development teams are still focused on answering the question, “Are we done testing?”

Fundamentally, this is the wrong question. It ties the concept of “quality” to static tests that produce multiple, independent and primarily binary data points of pass or fail. This approach results in a lot of data points, but not the information needed to help the business understand the real impact to the end-user experience.

(Related: Putting the test back in DevOps)

Understanding the specific risks associated with each release candidate becomes mission critical as organizations attempt to accelerate the release cycle. Without this visibility and knowledge of the impacts to the business, managers are unable to make the appropriate tradeoff or timing decisions for releasing software.

Instead of “Are we done testing?” we should be asking, “Does the release candidate have an acceptable level of business risk?” This new question is much more complex than it seems at the surface. It carries a few critical assumptions:

  • The inherent business risks associated with a given application and the particular release candidate are well defined.
  • There is an understanding of how to measure each of these defined business risks.
  • A baseline and thresholds are established for defining what constitutes an acceptable level of risk. Some business risks might have zero tolerance and no thresholds for acceptance.
  • Automation is in place to continuously assess the state of the application versus these defined risks.

This is why the concept of Continuous Testing is so critical. Continuous Testing provides an automated, unobtrusive way to obtain immediate feedback on the business risks associated with a software release candidate. It balances the traditional bottom-up tasks associated with software development and testing with a top-down approach focused on safeguarding the integrity of the user experience while protecting the business from the potential impacts of application shortcomings. Given the business expectations at each stage of the SDLC, Continuous Testing delivers a quantitative assessment of risk as well as actionable tasks that help mitigate risks before they progress to the next stage of the SDLC. The goal is to eliminate meaningless activities and produce value-added tasks that drive the development organization towards a successful release.

Continuous Testing is not simply more test automation… nor is it a “plug-and-play” solution. As with all process-driven initiatives, it requires the evolution of people, process and technology. We must accommodate the creative nature of software development as a discipline, yet we must face the overwhelming fact that software permeates every aspect of the business—and software failure now presents the single greatest risk to the organization.

Continuous Testing (when executed correctly) provides four major business benefits. First, it results in clearly delineated business risks associated with each application in the organization’s portfolio, including measurement standards for assessing the level of risk. It guides business and technical teams to collaboratively close the gap between business risk and development activities.

Second, Continuous Testing establishes a safety net that allows software developers to bring new features to market faster. With a trusted test suite ensuring the integrity of the related application components and functionality, developers can immediately assess the impact of code changes. This not only accelerates the rate of change, but also mitigates the risk of software defects reaching your customers.

Third, Continuous Testing allows managers to make better tradeoff decisions. From the business’ perspective, achieving a differentiable competitive advantage by being first to market with innovative software drives shareholder value. Yet, software development is a complex endeavor. As a result, managers are constantly faced with tradeoff decisions in order to meet the stated business objective. By providing a holistic understanding of the risk of release, Continuous Testing helps to optimize the business outcome.

Fourth, when teams are continuously executing a broad set of tests via “sensors” placed throughout the SDLC, they collect metrics regarding the quality of the process as well as the state of the software. The resulting metrics can be used to reexamine and optimize the process itself, including the effectiveness of the tests. This information can be used to establish a feedback loop that helps teams incrementally improve the process. Frequent measurement, tight feedback loops and continuous improvement are all key DevOps principles.

To explore how Continuous Testing accelerates the SDLC, promotes innovation and helps mitigate business risks, we recently published “Continuous Testing for IT Leaders.” This book is written for senior development managers and business executives who need to achieve the optimal balance between speed and quality with software applications that are the primary interface with customers… and ultimately revenue.