The ability to ensure applications deliver consistent, responsive performance at all times is critical for pretty much every organization, and is especially vital for retailers and other e-commerce providers. 

Even if an app delivers the best, most innovative functionality, it won’t matter if loading or transactions take too long. Further, as users continue to grow increasingly impatient, the definition of “too long” continues to shrink.

As a result, it is critical to validate application performance, both before new services are launched and before periods of peak activity. For example, for a retailer, it is vital to do extensive performance testing in advance of Black Friday. 

Traditionally, teams have been doing what we’d refer to as “classic” performance testing. Through this approach, teams do end-to-end testing based on high volumes of virtual users or synthetic transactions. Quite often, this type of performance testing is likely to delay new releases by several weeks. Further, this type of testing is extremely costly, invariably consuming a lot of staff time and resources. These challenges are particularly problematic in modern continuous delivery lifecycles, where keeping cycle times to a minimum is critical. 

To combat these challenges, it is imperative that teams gain the ability to speed software releases, while consistently ensuring they’re not introducing performance issues into production environments. 

Introducing Continuous Performance Testing

While a lot has been written about continuous testing, much of the focus has tended to center on continuous functional testing. A subset of continuous testing, continuous performance testing (CPT) is based on the principle of “continuous everything” in DevOps. As opposed to having a single performance testing phase, CPT is employed in situations in which performance testing needs to happen across different phases of the continuous integration/continuous delivery (CI/CD) lifecycle. 

CPT is a key enabler of continuous delivery. With CPT, teams can ensure apps are peak-performance ready at all times, so they can release new code without lengthy performance testing delays. There are three keys to enabling the successful implementation of CPT: 

  •     Teams must be able to specify performance requirements at the component level. These requirements must either be tied to functional requirements or product features, or they need to be tied to an application’s specific system components.
  •     Teams must be able to test each component in isolation.
  •     Teams have to be able to test frequently as application changes occur. 
Continuous Performance Testing: Three Best Practices

As organizations seek to employ CPT in their environments, there are a few key practices that will help ensure implementation success. Each of these practices is detailed in the following sections. 

Test at the Lowest Possible Level of Granularity

With CPT, most testing can be done at the unit, component, or API levels. By establishing testing at the component level, teams can test early and often. This component-level approach offers advantages in speed and operational efficiency.  

Another key advantage is that this approach reduces the number of tests that have to be completed: If component-level tests don’t pass, teams don’t need to run higher-level tests. This means teams can reduce the amount of resource-intensive, end-to-end tests that have to be executed. This also means more testing happens at the CI level, rather than at the CD level, where minimizing lead time for changes is most critical. 

Establish Frequent, Change-Driven Testing 

Once teams begin to do more granular, component-level testing, they can then employ another approach that helps reduce elapsed testing time: change-impact testing. Through this approach, teams can focus testing on specific parts of applications that have changed. At a high level, there are two ways to make this happen:

  •     “Inside-out” approach. In this scenario, teams take an inside-out approach by analyzing the impact of changes made in the code of application components. 
  •     “Outside-in” approach. In this case, taking an outside-in approach refers to focusing on analyzing the impact of changes made to application requirements or behavior. Through this outside-in approach, every time a requirement is changed, teams flag the set of tests that have been affected. In many organizations, this approach has reduced the amount of ongoing testing that is required by approximately 70%.
Scale Testing of Individual Components 

As mentioned above, doing end-to-end performance testing is expensive and time consuming. Through CPT, teams can effectively scale testing on specific components and reduce their reliance on these resource-intensive tests. To further scale component-level testing, teams can integrate CPT activities with CI/CD orchestration engines. In this way, teams can automate a range of efforts, including provisioning of environments, deployment of app components and test assets, execution of tests, capture and dissemination of test results, and post-testing cleanup. Teams can also leverage continuous service virtualization and continuous test data management, which can further boost scalability and test coverage. 

To learn more, view my earlier article entitled, “Optimize Continuous Delivery of Microservices Applications with Continuous Performance Testing.” This article features step-by-step guidance for implementing CPT across all the stages of the CI/CD pipeline.