The Hidden Costs of Testing

Even though testing consumes an average of 50% of IT projects, I’m willing to bet there is not a line item in most corporate budgets called “testing”. While those that develop software internally may have a QA department that is budgeted for, those that license packaged applications probably don’t because test resources are borrowed from the business or other tasks. As a result, the cost of testing – which is substantial – is effectively hidden.

Even worse, I am certain there are no companies whose financial statements have a line item called “inadequate testing” or “software failures”. Yet the evidence is irrefutable that the cost of failures in production can be significant or even catastrophic. There are direct costs such as increased support or helpdesk demands, rollbacks, re-testing and emergency transports, but also indirect costs in the form of lower productivity or even reduced revenue or lost customers. Then there are the ripple effects – resources that are diverted to handle emergencies are not available to complete planned projects, resulting in an overall slowdown.

It’s not that these costs are absent, they are just hiding in plain sight as increased operating costs, reduced revenue or market share, and overall decreases in productivity and profitability.

So why does this matter? For the simple reason that if you can’t measure it you can’t manage it, and without an accurate accounting of costs management can’t make informed decisions about the drivers that affect testing and all the related effects of trade-offs.

Maybe it’s my inner accountant that bristles at the idea of unacknowledged expenses, or maybe it’s the recent hoopla about how transparent the federal budget is or isn’t, but no matter what the reason I think it’s just plain wrong to essentially ignore such a critical business driver.

The Cost of Widgets

If a company is making widgets, the generally accepted accounting principles (GAAP) that must be satisfied for a clean financial audit demand a detailed cost accounting of all the components that create the widgets: direct variable costs such as materials and labor must be tracked and indirect fixed costs such as supplies and overhead must be allocated. And this is rightfully so: otherwise, how can the company determine if it can make widgets at a profit, or at what production level it must operate to breakeven?

This approach to accounting for costs enables informed decisions. For example, the decision to invest in automation equipment on an assembly line can be objectively evaluated by comparing the costs of performing a particular task manually versus automated, then calculating how many widgets over how much time would be required to recoup the cost. Throw in the cost of capital, estimate a useful life for the equipment, and viola: you have a dispassionate financial analysis of the return on such an investment.

The Cost of Software

When a company makes or buys software, these rules are strangely absent. The only GAAP rules center around whether the costs should be expensed or capitalized, based on the useful life of the application. But the detailed tracking of cost components is missing. At best, companies might track the project timeline by task, but few if any can report with any accuracy how much it costs them to analyze, design, develop, test, implement and maintain their applications.

This lack of accounting for software costs disables informed decisions. For example, I asked the CIO of a global company how much was spent on testing during their last upgrade. He estimated “A lot…at least 20%.” The actual cost, according to the project manager, was closer to 70%. This level of misinformation makes it impossible to adequately evaluate decisions such as the automation of testing. It is especially frustrating when executives ask for a ROI analysis for test automation yet can’t supply the most basic of metrics about their existing cost structure.

The Cost of Ignorance

In this case, ignorance is not bliss. It leads to poor decisions that result in higher costs, lower revenues and decreased profitability. There are even cases – some of them famous – where inadequate testing literally bankrupted the company. At a minimum, the hidden costs of testing mask forces that affect key business drivers and cripple effective management of scarce human and financial capital.

So how good is your software cost accounting? Take this quick test to find out:

· Of the total cost of your last project, how much was allocated to each task of the application life cycle?
· Do you account for the time spent by business users and other borrowed resources in doing testing? What was the opportunity cost of other tasks they could or should have been doing?
· How many emergency transports did your company perform last year, at what cost? What was the cost of delaying planned projects as a result of these unplanned activities?
· What was the impact on revenue and/or costs of software errors or lack of availability?

If you can’t answer these questions, the cost – and value – of testing is hidden from view, and as long as that is true you won’t be able to make the best decisions about an activity that consumes as much as half of your IT projects. And by any measure, that’s significant.

The Cost of Doing Nothing

You might agree in principle that better information would be helpful, but since you’ve managed all these years without it, what’s the rush? Why do anything different, especially now? Precisely because this is the best time to get smart about costs. When the economy is unpredictable – some might say unprecedented – it is time to change your thinking.

As Yvonne Genovese, vice president distinguished analyst with Gartner Research, points out: “We find that automating testing has a very compelling argument in that it’s important during the downturn to help us get more efficient – and we are in dire need of ways to cut costs in our current IT environments,” she said, “Manual testing is very inadequate, expensive, and introduces risk in the environment, where automated testing will help us take advantage of uncertainty and position us for the future because it allows us to test –and validate--across the end-to-end of the process, capture the test for reuse, and document the process, all while freeing up valuable resources.”

Exposing the hidden costs of testing may reveal opportunities for advantages that will pay benefits far into the future. There has never been a better time.