How to approach the risks
The process of testing in production depends on the application itself and what is being testing. Several items need to be in check in order to mitigate risks, no matter the testing environment. Businesses could experience a loss of transactions or commingling of test data with production data, and experts agree that a major risk of testing in production is the business risk. A bad user experience, security issues or system crashes could all lead to a loss in profit or tarnish a brand.
Among the risks are:
• Exposing potential vulnerabilities to the public
• Loss of data
• Poor user experience that damages the reputation of the organization
• Relying on users to report defects and vulnerabilities. Many will simply leave the app and not use it again, rather than filing a bug report with the organization
“The best way to mitigate the risks is by taking the right approach and by thinking about the outcomes,” said Margo Visitacion, vice president and principal analyst at Forrester Research. “It’s not about building the features and not just did you deliver on the user end, but did you consider performance and data integrity and test processes?”
Neotys’ Stoner said that measuring an application’s performance in the production environment alone can be a risk, and before launching a test all consequences should be considered.
“There’s always a right way and wrong way to get started,” he said. “Taking what you are doing in lower environments, be it pre-production or staging, and trying to execute that in production can be problematic.”
Before figuring out how to get started, professional testers need to begin by asking this question: “What do we mean by production?” as Rex Black, president of RBCS, states, because there are various processes for multiple production environments.
“You might be testing in production in one production environment—usually a smaller-scale environment—prior to deploying to the (typically larger) other production environments,” he said. His clients also use this kind of testing in production for their beta tests.
Black paraphrased the cons of testing in production with a family-friendly line from “Body Heat,” said by Mickey Rourke’s arsonist character: “There are dozens of ways that testing in production can go wrong, and if you think of half of those ways in advance, you’re a genius.”
TiP in an agile environment
Testing has long belonged to testers and QA. With agile, that has changed: Developers are now often asked to perform more of the testing, and testing can occur on a weekly, daily or hourly basis. In an agile environment, companies could test on every change on code, so every time the software evolves, every change of code is tested, according to Alon Girmonsky, CEO of performance management company BlazeMeter.