Not all organizations face the same business risks associated with application failure, and the cost of software quality certainly varies across industries. Remember: The cost of quality isn’t the price of creating quality software; it’s the penalty or risk incurred by failing to deliver quality software. Given that, one thing that doesn’t vary is the fact that organizations that test in production don’t necessarily advertise that they’re relegating a large part of their quality process to unsuspecting customers.
All too often, application updates in a continuous-delivery process are tiered. Updates are premiered at the lowest tier, using the lowest-priority clients as guinea pigs who unwittingly serve as real-time user-acceptance testers. If this real-time user-acceptance testing doesn’t indicate any major problems, the updates are then pushed out to higher-value customers.
However, if the “early adopters” report significant issues in the updated version, the organization rolls it back, tries to resolve the problems, and then starts the tiered process all over again. The organization recognizes that they are putting a certain percentage of their users at risk, but they consider this a necessary evil for getting the release out into the field.
Do you remember when the media exposed the use of “pink slime” as a meat additive? If not, you can see this video for a refresher course on all the grizzly details. Organizations leveraging their unconsenting and unaware users as QA are the equivalent of the meat industry using pink slime: It’s an unpleasant business reality that they would prefer to keep hidden.
Although the use of pink slime might have been a “cash cow” for the beef industry for many years, the exposure of this practice has already driven at least one beef producer to bankruptcy and forced the industry as a whole to think long and hard about whether the business risks are truly worth the cost savings. Likewise, backlash stemming from the current media spotlight on software failures—both functional glitches and security breaches—is starting to force our industry to reassess the true cost of quality for software.
Rollback as a quality strategy undeniably places the user experience at risk. With application switching costs at an all-time low, you’re really opening the door to defection caused by poor user experiences. For example, consider the recent proliferation of Yahoo Mail frustrations that reportedly prompted a mass migration to Gmail and other providers.
Moreover, if the users are actually paying subscribers/customers, it’s even worse: You’re forcing people to pay you in exchange for the “opportunity” to serve as your guinea pig. Once some lower-tier paying customers discover that their dollar is not as valuable as other people’s dollar, you’ve got a big problem.
All this being said, using your customers as real-time user acceptance testers can be a business strategy. However, if software professionals decide to take this route, we really need to ensure that two minimum requirements are met. First, the business must overtly recognize that this practice is part of the organization’s overall execution strategy. Second, the business must truly understand the potential risk or cost of quality associated with this practice.
From my experience, most organizations experience a wide gap between development’s technical decisions and the overarching business drivers that concern executive management. If both the business leaders and the development organization agree to use pink slime, there’s no problem: It’s a business decision. However, if management believes that the organization is providing 100% grass-fed, free-range, all-natural Angus ground beef with no filler—but development is in fact distributing beef laced with an ammonia-treated mash of meat trimmings—you’re undeniably opening the door to problems somewhere down the road.
Now that software has morphed from a business process enabler into a competitive differentiator, business expectations about the speed and reliability of software releases have changed dramatically. With a perfect storm of all-time-low switching costs, downward price pressure, and relentless media coverage of application failures, software quality matters more than ever. Ignoring this sea change regarding tolerance for faulty software is now a tremendous business risk: a risk equivalent to McDonald’s resuming the use of pink slime now that the public is all too aware of what that entails.
Wayne Ariola is Chief Strategy Officer of Parasoft, where he leads the development and execution of the company’s long-term strategy.
