A friend of mine, who is an experienced venture capitalist in the tech sector, once stunned me by claiming, “You need two people to write great code. One to program, and one to shoot the programmer when he thinks that he is done.”
At the time, this seemed to be a little extreme… but…
Today’s software, particularly business software, is fundamentally complex. Enterprises have wide-ranging requirements, demand many different features, and require constantly evolving functionality. On top of this, businesses need their systems to be flexible, and to provide their users with a host of options.
Hasso Plattner, the former CEO of SAP, once responded to the question posed by one of his clients, “Why is your software so complex?” with the remark, “If you were prepared to make your business less complex, then I could make my software simpler.”
It is challenging to construct software that supports all the features desired by businesses, spanning multiple functions, industries and geographies, in one product. When you add in the need to refine and refactor code, as well as to keep up with changing regulations, technology and trends, it is sometimes surprising that any usable business software ever gets shipped. Hence the sentiment behind my friend’s observation: The job of a programmer is never-ending, and to ship a product, there needs to be some external mandate.
Of course, it is one thing to finish development, but it is quite another to release finished software. There is not a developer alive who writes flawless code, so the completed code needs to be tested, which prompts the question: Just how much testing is needed? When I was once a pup programmer working for a U.K. defense contractor, the government auditors would check our code by counting up the number of conditions (i.e., the “if” statements), and check that at least one formal test condition existed for each one.
Today, at iBE.net, with our codebase extending toward 1,000,000 lines of code, applying such a rigid test methodology is both impractical and inconsistent with agile. Just within our mobile app (50,000 lines of code), there are some 3,500 conditions (“if” or “case” statements). Assuming a rate of 1 test per second, it would take 23,500 seconds to exhaustively test all conditions just within our mobile app. To test all the code, with all its combinations and permutations, would take a longer time than the universe has currently been in existence (which, incidentally, is estimated to be approximately 4 x 10^17 seconds).
#!
In my career, I have worked with a number of clients in the aerospace sector who remained incredulous about the quality of software shipped by leading ERP providers. To quote one CIO of a global aerospace company: “If I made my product the way those guys write software, there would be a line of people around the block waiting to sue me.” The criticality of an order-processing system is not the same as a life-threatening flaw in a jet engine; but the principle is correct: Any product should be expected to work as advertised, and that means it should be “adequately” tested.
As with jet engines, software is tested in a controlled environment. Testers’ access is restricted, and the data used is constructed to test as many features of the software as possible. It is very important that the test environment remain segregated, otherwise life gets interesting. One of my favorite testing urban legends, which allegedly occurred while I was working at a major consultancy in the U.K., was that test data “leaked” into the production system, and that led to a letter being sent to a customer saying:
Dear Sir,
Thank you for your letter dated <date>. Unfortunately, your claim is being denied for the following reason:
i) You are a fat b******
Now, I personally never saw the letter (or the response from the irate customer that went around the project team like wildfire), but I am assured that it is true.
At iBE.net, three months of exhaustive end-to-end scenario testing, normally reserved for test plans used by Fortune 100 companies implementing large enterprise ERP packages, is coming to a close. We are building automated regression test scripts using Selenium. This will allow us to aggressively roll out enhancements, leveraging the power of our SaaS based platform. Now we can start to welcome our beta customers onto our system. Will there be bugs? Almost certainly, but we hope that we caught the outliers before reaching any of our beta customers, through our extensive internal testing.
Of course, we will not be shooting any of our programmers, even when they claim to be done. But then again, if someone allows test data to leak into our production environment calling someone a fat b******, you know, we just might….
Simon Hopkins is cofounder of iBE.net, which offers cloud-based ERP software.
