Development in today’s agile world results in smaller, more frequent releases. According to Black, the smaller size of the release would mitigate some risk, but the increased frequency of releasing raises the risk of vulnerable code being put out live.
To help with testing in production, companies that adopt an agile workflow need to adopt an agile mindset, according to Forrester’s Visitation. Doing so will cut down on the possible risks of testing in production.
“You have to have the right product and the right attitude of product owners and Scrum masters, and a development team where they are really thinking about test-driven development,” said Visitation. “So they are thinking about what is going to be the functional performance of the features that are being delivered, what is the risk in security, elements or the features that are going to be delivered in the application as a whole, and really planning for that in order to be able to identify what needs to be tested and what needs to be automated.”
She said that in an agile environment, it’s important for an agile team to discuss and figure out what are acceptable risks and what are not, because in this environment, “You’re going to be accepting that, in order to get things done faster, you may not be getting everything in terms of functionality.”
Unintentional testing in production
Some say testing in production is akin to skipping testing altogether, in that pre-deployment testing will catch errors before they get to the application’s users. However, testing in production should not be done as a way to spend less money or do less testing.
“When that (latter) scenario does occur, it’s not referred to as testing in production,” said Rex Black, president of RBCS. “It’s just the very risky way software is put into production.”
Anytime software has been released without being tested, or it’s released thinking adequate testing was done, there is a chance that business-impacting incidents can occur. Black said that when this happens, and the company finds a bug in production, they were “unintentionally” testing in production.
Real-life examples he cited of problems occurring after release in software that wasn’t thoroughly tested are when T-Mobile had a loss or corruption of data, leading to a permanent loss of customer photos.
An even bigger issue was the memorable launch of HealthCare.gov, where serious security, reliability and performance problems occurred in production because the software was not tested thoroughly.