Recently, a simple conversation I had analyzing the current challenges associated with software testing in the modern era led to a key realization: the tools in the software testing industry have not been focused on simplicity for the Agile world.

Agile is primarily a development-focused activity. In its most basic terms, Agile is a software development methodology where typical SDLC activities that would traditionally span over the duration of a project are broken down into much smaller pieces called sprints. Typically, a sprint is 2 to 3 weeks and in a sprint, development activities are focused on new features and enhancements.

One sprint looks something like this:

A sprint starts with the design and creation phase, where the new functionality is split up into user stories, scoped, and then development immediately starts building something. At the end of the sprint, there may or may not be release activities, but no matter what, feedback is obtained and then another sprint begins and the process repeats itself over and over again.

Agile allows organizations to turn on a dime because the feedback collected during each sprint can be applied to the next sprint and help to guide, shape, and focus the project. This works great for development, but if you look at the test portion of the sprint, it starts to get complicated.

Test does not get access to test the new features and enhancements until well into the Sprint, and for logical reasons. The testing team needs to wait until the development team has built the full functionality, so test is always a little bit behind development right from the beginning.

Test can’t keep pace with development

This problem only intensifies as the sprint continues, due to the most common testing technique used to validate the application, which is by manually interacting with the user interface. This is known as UI testing and it’s the most common testing practice because it’s easy to use – it’s easy to associate actions in the UI with the user story, it’s easy to scale across a large body of testers, and because of record and playback functionality, it’s easy to do an initial round of automation.

But there are many problems with UI testing:

  • There are hidden costs that stem from the inefficiency of UI testing. The most fundamental challenge with UI testing is the amount of time it takes for development to fix defects when given a UI test as a reproduction. Typically, as a tester begins the defect discovery process, they start with exploratory testing (methodically searching the application searching for unexpected behavior). When they find a defect, they need to reproduce that for development, which involves writing up “steps to reproduce.” When development receives these instructions, they need to find the version of the application that test is using, stand it up, and go through the steps exercising the UI. If the defect reproduces, they then have to associate that defect with the underlying code to determine the root cause. Development starts working on a fix which requires them to tear apart the application, fix the defect, and then re-build the application before QA can start testing again. This further delays the software delivery process and slows down the whole pipeline.
  • UI testing doesn’t comprehensively test an application. Testing at the UI layer validates an application’s process flow end-to-end, but doesn’t necessarily test the entire breadth of the system’s internals. Often when new functionality is introduced into an application, it requires changing or updating existing interfaces. Some of these components may not be accessed when using the new functionality but present significant risks to the organization if they are not tested.
  • Test doesn’t have access to the code so is difficult for them to map the actions that they are performing in the UI to the underlying source. As a result, the tests that get built do not provide complete API coverage. Quite often things get missed. When it comes time to run the full regression cycle, critical defects may be uncovered. Often, this late cycle defect-detection leads to significant release delays and raises the total cost of testing.
  • It’s difficult to maintain UI automation. A main reason why test struggles to keep pace with development is because too much time is spent managing broken UI tests. In fact, up to 80% of testing time is spent either manually executing the UI tests or fixing automated UI tests that have broken as result of application change.

Each one of the above factors can lead to significant delays in a sprint, but when you take into account how a traditional project cycle works, it’s a series of these sprints followed by a hardening or regression cycle. At each step of the way, test is struggling to keep pace with development – but because of the test techniques traditionally used, they can never get the total and complete comprehensive testing that they want.

Typically, they will be able to validate the new features and capabilities, but fall short of complete test coverage.

This is frustrating for a lot of testers, but it’s not their fault. It is just the nature of the beast given the capabilities that exist in the tooling market. The dangerous part is that without these quality practices, defects are leaking into production and eroding the perceived benefit gained from Agile.

What about API testing? Can it save Agile? 

The analysts and industry agree that API testing can more precisely pinpoint the root cause of defects than UI testing because API tests are closer to the code, easier to automate, and more resistant to application change. Also, API tests offer a better form of defect reproduction and communication between development and test because the test artifact represents the convergence of those two areas.

In a recent blog I explored API testing, what it is, and how to build a comprehensive API testing strategy. You can read it to get more information about this extremely effective testing practice.

Testing at the API layer is a great practice for Agile specifically, because it enables testers to validate functionality given the compressed timeline, and API tests are highly reusable. Additionally, API tests have the following advantages:

Lower time-to-defect remediation when compared to UI tests 
If an API test fails, you can be pretty darn sure you know approximately where to look in the code. Developers love getting API tests from test because they can execute them directly against their application without having to hook up the entire environment. And they can continuously rerun them as they are starting to fix the defect.

The lower time-to-defect remediation means that in general, development can fix a bug faster when provided an API test vs UI test. When considering the time frames involved in Agile, this is exactly what we need. Once a defect is discovered, an API test is provided to development, which they can use to find, fix, and validate the defect, all without having to rebuild the entire application, which saves a tremendous amount of time. This is exactly the kind of speed we need for Agile.

API tests are “automation ready”
APIs represent the invisible communication that takes place behind the scenes of an application. The invisible nature of the communication helps the automation process. There is significantly less complexity involved in getting an application to the point where you can start interfacing with it at the API level than would be required to stand up an application in its entirety so you could operate at the UI level.

As a result, API tests can easily be run in automation at earlier stages of the SDLC. Most of my customers run them at the same time they are running their unit tests as a function of code check-in. These API test runs can also be associated with the bug tracking system in a much easier way so that when defects are resolved, the accompanying API tests can easily be handed back and forth between development and test. This significantly reduces the overall handoff process because instead of filing the defect, providing the steps to reproduce, and then waiting for a new build from development, testers can receive notifications from the bug tracking system that a defect has been resolved and see the automated test cases that validate the resolution. Those API tests can easily be built into a regression suite and reused over and over.

API tests are more resilient to change than UI tests
As a part of our research, we saw that 80% of development time was spent on managing and updating UI tests that had broken as a result of change. Change is a big time killer when it comes to Agile, but because of the increased code check-ins and shortened time frames introduced with Agile, change is constant.

If an organization has exclusive reliance on UI testing, application change can be devastating because many of the test cases that have been built to validate critical functionality simply stop working. One of the main principles of Agile is the ability to turn on a dime, which means that UIs and functionality are changing all the time and the burden of supporting and maintaining those tests can become overwhelming for test teams. On the other hand, API tests don’t even see the UI. APIs also have specific versioning capabilities built in, that allow testers to maintain stability as the application is undergoing change. Additionally, APIs are defined with the service contract that can be leveraged to update test cases as the application undergoes these changes.

API tests can save agile by giving an organization the ability to easily test an application at the earlier stages of development as well as provide an effective communication mechanism between development and test that is highly resistant to change. Organizations that adopt API testing as a fundamental piece of their testing strategy can leverage the agility they provide to really get ahead of the testing challenges.

So why aren’t organizations API testing?

Even with all of the benefits that come from API testing, the industry is still focused on UI testing. We believe this is because testers don’t know how to test the API and/or don’t know how their application is using the APIs. It’s not immediately apparent where to get started API testing an application, and understanding how to assemble all of the “puzzle pieces” together in a meaningful way requires domain knowledge of the application. Since organizations still tend to leverage a centralized testing practice, testers need intimate knowledge of all the different application interfaces and know how to stitch them together properly. It’s not a trivial task.

API testing is still considered no man’s land. In a recent survey, we asked a series of developers and testers who is responsible for API testing in their organization.

  • 70% of testers said that development is responsible for API testing. (“The development team created the APIs and as they were built they should have also built API tests to validate that they work as described.”)
  • 80% of developers said that test is responsible for API testing. (“We created these APIs to be externally facing and documented them with a service contract. The testing team should come in and validate that the APIs work as described.”)

As you can see, there’s some confusion as to who is ultimately responsible for API testing. I believe that API testing is the responsibility of both developers and testers, in different forms, but it is this disconnect that leads to low API test coverage.

Testing at the API level requires specialized skills and tools in order to get comprehensive test coverage. It’s not intuitive. There are tools that exist in the market that are trying to help organizations build an API testing strategy, but the vast majority of them require a high degree of technical expertise to build comprehensive API tests. Additionally, testers still need to understand how the APIs work, which requires domain knowledge. As a result, organizations tend to do the bare minimum for API testing, which is the opposite of what we need for Agile.

Artificial intelligence for test automation

Why AI, why now? The only way to solve this industry problem is to build tools that take the complexity out of API testing. New technology is using artificial intelligence to help take the complexity out of API testing by converting manual UI tests into automated API tests, lowering the technical skills required to adopt API testing and helping organizations build a comprehensive API testing strategy that scales.

So how does it work?

Converts UI activity into automated API tests
The technology monitors background traffic while you’re executing manual tests, analyzes that traffic, and uses artificial intelligence to automatically build a meaningful set of API test scenarios. When building those API tests, it first identifies API calls, then discovers patterns and analyzes the relationships between them, so it can generate complete API testing scenarios, not just a series of API tests.

Reduces the learning curve to API testing
With an easy place to start building API tests, testers don’t have to touch the difficult activities associated with manually building API tests, i.e. finding the correct service definition, understanding the data payload, or running a test over and over again to understand the relationships between requests and responses so you can start building assertions.

Instead, the AI-enabled technology does all of this heavy lifting automatically, based on activity it observes while the tester is using the UI. This helps novice users get a greater understanding of API testing in general because they can map the activities they performed in the UI to the API tests that were created, and build a greater understanding of the relationship between the UI and the underlying API calls, helping drive future API testing efforts.

Helps users build comprehensive API testing strategies
Although API testing is one of the most effective software testing practices, many organizations haven’t successfully adopted the practice because it requires specialized skills and tools. To help organizations adopt a comprehensive API testing practice, it is important to also use a sophisticated API testing tool that provides visual tools that are easy to adopt, enabling API testing beginners to start creating powerful API scenarios in a short amount of time.

What’s the end result?

Agile development helps organizations deliver quality software to market faster, but without needed technology to help organizations fully test their applications at speed, the risks associated with accelerated delivery erode Agile’s potential benefits. Now is the time for organizations to get smart about API testing.

A solid API testing strategy will allow organizations to get the most value out of their Agile transformations. To make this a reality, testing tools should work for us. Using artificial intelligence to do the heavy lifting of creating API test scenarios lowers the complexity associated with API testing, lowers its adoption barriers, and helps organizations bring in a manageable, maintainable, and scalable test strategy.