Let’s say that you have a function that takes two strings and returns a single string. Being a responsible 21st-century developer, you want to create a set of unit tests.
You certainly want to test the boundary conditions, when nulls are passed in. And certainly what if the strings passed in are empty? Oh, and what if they contain escapes? And what about their encoding? Oh, and what about…
Even the simplest function generally has a large number of corner cases that ought to be validated. True, managed code makes nulls and over- and under-flows and (heaven forbid) boundary violations less catastrophic, but, caught as a runtime exception or not, an unanticipated value causing an abnormal end to your program is a problem.
Most handwritten test suites skip over the boilerplate cases, because the test developer jumps ahead to “realistic” or “interesting” cases rather than spending their valuable time validating the almost-certain-to-pass simple cases. Of course, battling such “of course this part works” assumptions is key to unit-testing.
One tool to help get through the boilerplate corner cases quickly is FIT (Framework for Integrated Test), which reduces inputs and outputs to expected values in a table. Even more sophisticated, but requiring a larger commitment, are tools that support FIT-style tables in the context of Behavior-Driven Development. Cucumber is very popular in the Ruby world, and there are a number of .NET frameworks supporting the “Gherkin” language, such as SpecFlow.
Pex, a Microsoft PowerTool for Visual Studio 2010, is the first tool for white-box testing that I find compelling. It is not a brand-new tool—I discussed it briefly two years ago—and the PowerTool is nicely polished. I feel that the crucial benefit in Pex is that it works by analyzing the internal structure of the code under test.
Rather than exhaustively running every combination of possible inputs (impractical for non-trivial functions) or just using boiler-plate “interesting” values for the input parameters (min and max values, -1, 0, 1, etc.), it uses constraint programming to seek values that exercise all possible code paths in the target function.
The results can be deceptively simple-looking. Handwritten tests almost always have human-significant values: “InvalidEmail” and “Robert’);DROP TABLE Students;” and so forth. The output of running a Pex “exploration” has less obvious data, a FIT-style table with lots of single-character and “ ” strings.