As agile practices, Continuous Integration and other new methodologies come into development environments, organizations have come to realize they need to change the way they test software.
Among the changes highlighted in a recent survey of more than 500 developers by Dimensional Research and sponsored by Sauce Labs are that organizations are doing more automated testing at the top of the process, are testing earlier, and are doing more testing overall.
Of the respondents, 94% said testing has changed since their organizations adopted agile and CI, while 69% said they’re doing more automated testing, and 61% said they’re testing earlier in the development cycle. The respondents were mostly developers, as opposed to QA professionals, according to Diane Hagglund, founder of and senior researcher at Dimensional Research.
She said that even with a renewed emphasis on test by developers, they “are not writing better code, but all this testing is catching the errors.” Only 17% of the respondents said that agile and CI have helped them write code with fewer errors.
The study found that across the gamut of test types, more testing is being done in agile shops than in non-agile shops—with the exception of manual testing. Eighty-nine percent of non-agile shops are doing manual testing, while 68% of agile shops are doing it.
The study also found that there is no clear consensus on how to test applications for mobile platforms. A little more than a third of respondents said they test on real devices only, while a third said they use a combination of real devices and emulators, and a little less than a third use simulators or emulators only.
And, of the third that use a combination, about a third use mostly real devices, about a third use devices and simulators the same amount, and less than a third use mostly simulators and emulators.
“This is a classic example of a market not being sure of the best way to do this,” Hagglund said. “There is zero agreement among testers on how to test across mobile platforms. There is no best practice; they’re not near there at all.”