Tips for TiP
Load and performance testing bring along many challenges, and one of the biggest is the testing environment itself, because most companies do not have the budget to recreate an environment that reflects their production environment. Given the challenges, Brad Stoner, senior performance engineer at Neotys, and his colleagues (namely Tim Hinds, product marketing manager, and Henrik Rexed, performance testing specialist) have broken down some of the confusing components. They’ve given some tips that you should consider before testing in production.

Timing of tests: Stoner said studies show that there is a link between user experience and profit, depending on the application. Running a load test during business hours increases the chances of a bad user experience, so he recommends running tests on the production environment during times that impact fewer real users. This includes during the night (or non-business hours), after deploying a new release, and during maintenance hours. It’s a short window for the performance engineer, but these are the timeslots that will have a lower impact on users.

Know the challenges and environment: Never launch a load test on a production environment without knowing anything of the released application’s level of performance. Stoner considers this a big risk, a risk that could even get you fired. Testing environments can differ in ways like the amount of servers, types of network equipment, integration with third-party tools, or utilization of a content-delivery network and bandwidth by other users or applications. Even with testing the application in a single environment, they recommend considering several resources (in addition to the performance engineer, who will be ready to react should something impactful occur):
• Operations: They can unplug the current production environment for the test.
• Architect or technical leader of the project: The guys who look at logs to identify any potential issues.
• DBA: They manage stability, and will find “blocking points” on the database and will “replug” the production database.
• Project leader: The person who will define timeslots for load testing, and inform users of potential disruptions.

Monitoring: In large organizations, testers might not have access to the operations team’s monitoring tools, so getting visibility of what is going on in production is a challenge. Whether it is functional or performance monitoring, the Neotys group said you need monitoring to see what is happening on the servers or in the databases, and if there is a problem, you can figure out why. “If I didn’t have production monitoring, I would never test in production,” said Stoner. You could say that good monitoring is a requirement to test in production successfully.

Cause chaos on purpose: Causing chaos on purpose is a method to ensuring that production is working as expected, but Stoner says this method is something to be wary of. A “chaos monkey” is code that introduces failures on purpose or kills a Web server in production to cause problems in an environment. The point of it is, if you’re designing a highly resilient application, and the chaos monkey kills one of the tiers, you must be able to maintain levels of service.