Using agile development methods, continuous build and deployment, organizations can produce several builds in a day. But if it takes 30 hours to test a build, how do testers get it done by tomorrow? How do they keep up?
By leveraging the automation capabilities of the very tools that are threatening to bury testers, to be able to do more testing and increase test efficiency.
A couple of years ago, I was testing the customer-facing and internal-facing portals for a company that provided VoIP services. It took about 30 minutes for me to manually deploy a Git Tag to my virtual Unix/Tomcat test environment. After that, it took about 60 hours of testing to go through the major scenarios and validate the results.
Eventually I cut it down to 30 hours of testing by adding some automated scripts to prepare the database with the test accounts and profiles that my test scripts needed. This saved me hours of manual data entry in the portal interface. I also added some automated scripts to help quick-test the basic portal features, but the real testing had to be done manually.
Sales and Marketing were evolving the “perfect” feature package plans. As a result, the customer’s billing statement changed with every new build. The requirements were fuzzy and constantly changing as they tried out different ideas.
Also, as we neared the release date, I discovered that certain rapidly proliferating inventory items in the production database could not be deleted when they ceased to exist in the real world. So, we started testing database fixes that could not be completely verified in testing due to the differences between the QA and production environments, meaning it would have to be tested again in production.
In this same timeframe, unscheduled and untested changes were made to some IP tables during the weekend. The resulting errors caused major outages in the customer’s phone services. The director called a big meeting and told everyone to slow down and be more careful. He said, “If someone asks you to do something you are not comfortable with, raise your hand and say, ‘I am not comfortable doing this.’ ”
After several extensions to the release date (due to continuous changes in the application), Development decided to schedule the release to production. They told me I would be getting the release-candidate tag for testing the next morning (Friday), and they were going to deploy the release on Saturday. So, I could verify the release in production on Sunday, giving me a sleepless 24 hours to do my 30 hours of testing in QA, and another 24 hours to repeat the feat in production.
Given the risk to production, I wrote an e-mail saying the magic words: “I don’t feel comfortable doing this…” My contract was cancelled an hour after I sent the e-mail, and they released the portals on Sunday as planned (without my testing).
Lesson learned: Test faster
Everyone needs to get code into production faster. So, even if management seems sympathetic to the problem, the fact remains that we have to deploy and test faster, or get dropped by the wayside.