Continuous testing marks the evolution of software testing to where it can keep up with the rapid pace of development, while adapting to new software architectures and development methodologies such as DevOps and Agile.

Relying on automated test execution as a cornerstone, continuous testing is the embodiment of the “shift left” craze sweeping the industry. Move everything up earlier in the life cycle, so nothing is left behind. That means even at ideation, security people, business people and testers are considering their role in bringing that idea through to a final product.

But for those just tipping their toes into the DevOps waters, it’s often hard to figure out how to put the pieces together and where to begin.

RELATED CONTENT:
Why continuous testing is so important
Who owns continuous testing?
Hundred-year-old company prioritizes continuous testing
A guide to continuous testing tools

“Where I normally start is talking about what does that mean to them? It has some different definitions,” explained Mark Lambert, vice president of products at testing software provider Parasoft. “Continuous testing is, in essence, you’re building testing into your pipeline so it’s running continuously as you’re doing your development, and it happens automatically. It goes beyond automated testing but this is where somebody starts. Somebody may start by building some automated tests and running them as part of the pipeline.”

Mobile development creates its own challenges for organizations looking to begin continuous testing of their applications. Steve Orlando, senior director of product marketing at Mobile Labs, said you first should identify issues or bottlenecks. “Are you slow to release? Maybe you’re not getting fast enough answers from QA,” he said. “Once you identify these challenges, it’s important to set some goals. When you know what you want to do, such as improve turnaround time from testing or find bugs faster, it’s then important to find the right tools to aid your process and to get the job done. Look for tools in a build system, where you can continuously build and integrate your code and tests. When code is checked in, it can then be built and compiled for continuous testing.”

Brian Reed, chief mobility officer at NowSecure, describes three keys: First, catalog your applications and assign risk profiles, with the idea to test the high-risk apps at every build, every day. Next, plug security testing into your build tools, ticketing systems and vulnerability management tools. Finally, he said, “security and DevOps teams should review existing security processes and testing/repair criteria to create a tiered model that defines P1/P2/P3 service levels of what to fix when in a continuous testing program.”

According to Eran Kinsbruner, mobile technical evangelist at mobile and web application test provider Perfecto, test automation is absolutely critical to continuous testing. Building tests to execute inside your continuous integration pipeline automatically ensures that each time code is checked in, it’s tested, at a place and time when it’s less expensive and easier to detect than later on in the software’s life.

“Automation today, without even exaggerating, is the most important pillar of continuous testing,” Kinsbruner said. “Continuous testing, by its meaning, means you need to validate code changes as soon as possible to uncover risks and release faster. You cannot do that if you are tied into a lot of manual testing. If you want to deliver fast, you want to release fast, you want to integrate new code changes, and the only thing you have is manual testing, it won’t be continuous and will actually slow you down.”

Kinsbruner stressed that automation is not a one-time thing for each software iteration. Instead, he said, you need to continually build on the automation, lest you fall behind because much of your process remains manual.

Automating as much as you can in testing also is important because it can give you greater code coverage than manual testing can offer. But Mobile Labs’ Orlando said how much you cover with tests is less important than what you’re actually testing.

“Honestly, 100 percent code coverage is a lofty goal that I’m sure all developers and testers wish was easily achievable,” he said. “But, in reality I’ve found that it is most important to focus on the most used functions within your app. Sometimes certain functions are not able to be tested via automation – that’s where manual testing comes into play.”

NowSecure’s Reed said that when considering code coverage for mobile app security testing, you must examine the mobile app attack surface itself. That, he noted, “spans data at rest on the mobile device (DAR), data in motion between device and back ends (DIM) and code itself (including custom code and third-party included libraries). Traditional Static App Testing (SAST) looks at custom code before build and misses DIM, DAR and third-party libraries. Achieving 100 percent code coverage of the full mobile app requires dynamic testing (DAST) to actually install and run the app on a real mobile device in the real world.”

Reed added that NowSecure benchmarks have found that more than 60 percent of mobile apps in the Apple App Store and Google Play Store leak data via DIM, DAR and/or third-party code.

The mere mention of test automation as the first step toward continuous testing begs the question: Are we talking about automating test creation or automating test execution? The answer, in fact, can be both.

“If you would have asked me this question five or six years ago, it was very clear: You need to write scripts, and you need to create them in a development language such as Java, VBScript, C#, Python and so forth,” Kinsbruner said. “In today’s IT, there are  two ways of creating test automation. The first one is the traditional one, where you use the IDE, like Eclipse, IntelliJ and so forth, to write test automation scripts based on user stories or even functional specs. You go by the definition of the thing that you want to automate and you write code in JavaScript or other languages. That’s the creation.”

The second way, he continued, involves artificial intelligence and machine learning. “With the rise of AI and machine learning tools, you can add to this creation activity a new method, which is record and playback,” Kinsbruner said. “You can point your tool into a website or mobile device, and based on supervised algorithms for machine learning, you can actually create test automation without really writing one line of code.”

Continuing on the path
After test automations have been created and put into the pipeline, where do you go from there?

According to Parasoft’s Lambert, the next step is to take tests you’d normally created in the later stages of development and execute them continuously moving forward, during active and ongoing development. And that, he added, is  often a conversation with the move from waterfall to Agile, “or staged development into an agile-ISH.”

“I would normally do load and performance testing at the end just before release,” he said, “and what I’m saying is, you can now do that because your cycles are compressed; you can now do that automatically as part of your CI process; and yes, you’re going to do a more complete end-to-end full system verification as things start coming together.”

On the personal side, getting people with different backgrounds and different skill sets to work together is a big challenge to overcome. As Perfecto’s Kinsbruner said, “While you want to make everyone work together, not everyone is wearing the same T-shirt. Some come with a background in Java, some in Python, or C#, and if your technology does not match the different squads involved in the different activities, development and testing, then you’re starting to have some hiccups in the overall process. you need to assess and maybe try to even match the right people into the right dev team and feature and product to be developed, so they find it easier to implement test automation. Otherwise, you’re just causing delays and so forth.”

Meanwhile, organizations have to be careful not to rush into plans or make promises that aren’t close to their reality, meaning: Are they using technologies such as artificial intelligence, or frameworks such as Cypress.io? “People are talking about those things, but whether they really match the activities the team needs to do is something to be determined. Just because it’s a popular thing, you don’t need to rush and choose these frameworks,” Kinsbruner said.

Parasoft’s Lambert agrees that frameworks are critical, but suggests sticking with open source. 

“I’m an advocate of leveraging open-source frameworks where they exist. For web testing, I’m an advocate of leveraging Selenium. For testing in the Java world, it’s mostly JUnit. Leveraging a framework and an open-source framework that has a significant amount of adoption is valuable because you can then take advantage of what other teams have done. In the JUnit world, for example, there are stubbing and mocking frameworks … so you don’t have to reinvent the wheel. Frameworks are a fundamental instrument, as far as I’m concerned.”

Perfecto’s Kinsbruner suggests aligning people with the right skill set, the right product and the right testing framework, because the framework the organization chooses needs to serve multiple personas. “Continuous testing is an activity that comes to validate your code quality upon each code change. It can be done by developers, or testers, performance engineers or by many other people who are involved in the process,” he said.

What should you test continuously?
For testing mobile applications, the critical areas to test are those that provide its main functionality. Those, said Mobile Labs’ Orlando, are the ones you must continuously test. “What does your app do? What’s its purpose? If the app is intended to help travelers book a hotel room, then you need to continually test and make sure that the app is able to book a hotel room 100 percent of the time,” he said. “It’s important to nail the functionality of your app before you explore other areas such as network virtualization and other features that are present but not tied to the main functionality. Until you’ve pinned down your main functionality, these other areas do not require continuous testing.”

According to NowSecure’s Reed, “Continuous testing starts with the risk profile of the mobile app portfolio itself.” Most organizations have high, medium-, low-risk mobile apps matrixed by frequency of releases such as:

  1. Tier 1 high risk apps typically are the primary customer-facing business apps on iOS and Android that have significant customer personal information, perform critical transactions and contain significant intellectual IP where continuous testing with automated security testing of every build (possibly every day) should be the default operating mode. In addition, an organization might add regular pen testing certification for these high-risk apps.
  2. Tier 2 medium-risk apps may have substantial company information, HR and IP, perhaps used by employees only, that should be continuously tested based on what is typically a less frequent release cycle where builds might be weekly or biweekly.
  3. Tier 3 low-risk apps such as conference room booking, cafeteria menus, or parking, may not require continuous testing. In these cases,  periodic but non-continuous testing strategy is most cost and risk effective.

Parasoft’s Lambert suggests following the testing pyramid, created by Martin Fowler. 

“You should have a significant number of unit tests, typically developer-centric, that typically don’t require service virtualization because they’re isolated by definition,” he explained. “Then as you start moving up the testing pyramid and up the technology stack, API testing … and then you want to minimize your level of end-to-end UI tests. It’s not to say to eliminate them; you want to make them as efficient as possible, so you’ve got to worry about maintainability of those tests. And the higher up the pyramid you go the more complex the use cases are as well.”

NowSecure’s Reed said to start small on one mobile application and in one DevOps pipeline, and then incrementally grow the continuous testing capabilities. Next, he said, “Partner across security, dev and DevOps to define SLA and process agreements to optimize for flow vs. perfect security, with incremental improvements over time.” Follow that with a focus on automation and integration into your existing toolchain, and finally, look for automated testing tools that “provide complete SAST+DAST test runs in 15 minutes or less to ensure full security coverage while meeting the DevOps speed requirements,” he added.

CT best practices for mobile applications
Steve Orlando of Mobile Labs suggests four practices for effective continuous testing:

  • Set up a proper build system that stops on any error and provides a notification to all interested parties. This allows issues to be addressed immediately.
  • Integrate unit tests for the application into the build steps.
  • Upload applications to a mobile device testing platform so the application can be used by the testers.
  • Run functional automated tests against real devices by utilizing a mobile device testing platform. 

Model-based testing … Good or bad?
Model-based testing, in which you define what an application’s functionality should be and test against that, can be used to augment more traditional test automation.

“I define model-based testing where I’m more focused about defining a model or a business process that represents how my functionality should work,” said Mark Lambert, vice president of products at testing software provider Parasoft. “Typically, model-based testing works at the higher levels of the application architecture, so when I’m defining through a business use case, it runs into challenges when you start trying to apply it to code that is not developed with a model-centric view in mind.”

So, should you be doing model-based testing? Here’s Lambert again: “Embedded safety-critical systems many times have a lot of models already …  so therefore creating the test based upon the models is actually very logical. If you’re doing model-based testing for enterprise IT-type of applications, you’re trying to reverse-engineer the models, which sometimes can cause inconsistencies. But it also gives you a view of how the business flow should work. So there are pros and cons to it. I don’t view it as a good [method] for validating or building a regression against a business requirement, but I view it as a good way of fleshing out many different scenarios within the business use cases.”