Earlier in my career, when I was a software quality engineer, I led the evaluation, purchase and implementation of software tools. As a tool admin, I helped colleagues analyze a variety of tools and better identify best of breed stacks. The process that I followed lends itself well to evaluating a range of technical solutions properly.
With that in mind, I’ve developed this short step-by-step guide for others who may need to select tools in the future:
Identify key technical capabilities: I recently read “The Hard Thing About Hard Things,” by Ben Horowitz. In talking about hiring executives he makes two points that also apply to selecting tools. He noted, “If you’ve never done the job, how do you hire somebody good (for that job)?” When applied to selecting tools, this might translate to: “If you’ve never done the job, how do you select the best tool for the people doing that job?”
(Related: Automation is the key to software testing)
And the next one: “Write down the strengths you want and the weaknesses you are willing to tolerate.” This concept is important, because no single tool will be everything to everyone. It’s easy to fall into a trap of selecting a “just ok” tool because you are focused too much on choosing one that doesn’t have visible weaknesses. Don’t undervalue the strengths you are looking for.
To begin, identify the key technical capabilities that provide value to the end users, administrators and management. An abbreviated list of such capabilities might include the following: platform compatibility, third-party software compatibility, turn-key capabilities, configurability (point and click), customizability (coding), flexibility to conform to a variety of processes, ease of use, strength/size of user community, etc. Your organization will have additional capabilities.
Next you need to weight the technical capabilities. When creating this scale, place higher weights on those capabilities you deem most important. In some cases, we put up to 50% of the decision on a single capability, with some other extraneous ones at only five%. This helps you evaluate for strength rather than lack of weakness, as described above.
The graphic below shows an example of capabilities and their weight, for a total of 100%.
Remember that it’s in your best interest to be thorough as you weight these capabilities, so don’t skimp on time or process. I participated in a few reviews in which a cursory process resulted in badly weighted capabilities, which in turn led to the wrong tool being selected. We’ve all heard the old “garbage in equals garbage out” saying. It’s true here as well.
Identify and evaluate the tools: Employ a variety of sources such as industry events, publications, user communities and industry peers to investigate and select qualified candidates. In your effort to identify candidates don’t forget to look inside your company. What tools are you already using? You might have several candidates already in-house. You might find that this exercise grows beyond tool selection and gets in to tool consolidation if you currently own tools that need retired. If you end up with multiple tools you might need to provide guidance internally on which tool to use and why. But we can discuss those in future articles.
Consistency is critical to any tool evaluation. Do not evaluate one tool by reading the vendor’s website and a couple datasheets, then evaluate another tool by bringing it in-house for a multi-week evaluation. Figure out how you want to score each capability and evaluate each tool consistently. Some capabilities can be evaluated by reading datasheets. Others can be evaluated with a live demo and Q&A. Some capabilities you must see for yourself in your own environment.
Using a basic scale of 1 to 5, with 5 being the best, fill in a matrix (see below) to score each capability for each tool. You can even break out a capability in to several attributes. Below I’ve averaged the three attributes of capability 6 for a single score.
Score the tools and make a short list of tools with those capabilities: Now it’s time for some math. The weighted score is calculated using the following formula and values from Step 1 and 2:
Weight / 5 * Score = Weighted Score
For example, Tool A Capability 1:
25 / 5 * 2 = 10
Next identify the “keepers.” In this case we would short list the top scoring Tool B and Tool C. Tool A is on the cusp, because while it showed strengths in some areas, it fell far short in others. This is your shortlist of tools that meet the technical requirements you’ve identified with an emphasis on the strengths you desire.
Calculate total cost of ownership (TCO), ROI and soft benefits: The “keepers” are now evaluated for non-technical aspects, like total cost of ownership and ROI. Cost was intentionally ignored until this point. The “keepers” were evaluated and selected purely on technical capability without the bias of cost.
Price certainly is a factor in most organizations, so consider all factors that go into the total cost of owning each tool. This goes beyond the obvious upfront licensing costs. It can be tempting to consider Open Source tools “free” due to the lack of licensing costs but Open Source can be extremely expensive when factoring in all costs. Non-licensing costs can be maintenance and support renewal, the time and costs associated with training users and admins, how many admins are necessary to deploy, maintain and upgrade the tool, the skill set required for tool admins and developers, and the fully loaded cost of those people. In addition, investigate any additional hardware costs required.
Many benefits are obvious and relatively straightforward to measure. But be sure to include anticipated time savings because of the elimination of manual processes.
“Soft” benefits are more difficult to measure, but are very important to consider. For example, the company will benefit from improved team engagement resulting in happier employees, better traceability resulting in better compliance to industry regulations, and improved management reporting with accurate data resulting in better decisions?
And finally, you’ll want to make a judgment on the vision and future direction of each tool vendors. Is the vendor capable of becoming a strategic partner? Do they deliver innovation at a regular cadence? Do they have the capability to continue delivering solutions several years from now?
Present options and recommendation to stakeholders: Now that you’ve applied your ranking system to each tool and better understand the total cost of ownership and value for each, present the options to management stakeholders. Initially lay out your recommendation based upon the best technical solution. This is important, since the best technical solution is likely the one that will be most beneficial for your organization. Beyond this, it will likely lead to the most satisfied developers/practitioners, perhaps even affecting your ability to retain them. And we know how hard it is to find and retain quality programmers.
If the best solution isn’t the least expensive one, be prepared to share the business risks of choosing a cheaper solution that is less technical qualified. These might include loss of productivity, data corruption, dissatisfied staff and spending money on something that may not accomplish its purpose. Avoid the churn by choosing the solution that will actually help you accomplish your goals.
If you follow these five steps, congratulations! You’re likely making a solid business choice. When you take the time to properly evaluate your options, your organization will enjoy measurable efficiency improvements and will become an attractive place for quality people to work.
