Worksoft

The Value of Values

2010-12-31T09:53:00.000-08:00

As the year and decade end (or not, depending on how you think about the year 0), I have been reflecting on what has come before and what might or should happen next.

In my humble opinion, Worksoft has done a masterful job of making test automation feasible. That’s a credit to a world class product team as well as many years of experience from founders, employees and customers - not to mention robust infusions of capital. Until application developers learn to behave ;-) our automation may not always be a pretty process, but it works and works well.

The company has also produced new products at an astonishing rate, displaying productivity without sacrificing quality. Many of these new products enable the integration of testing into an overall change management process, making automation even more powerful and valuable. I find this all exciting.

But looking ahead, I realize that once you can easily execute as many tests as fast as you want, the challenge moves from how to test to what to test. What to test refers to the coverage matrix, which I maintain is the set of business processes and rule outcomes that should be verified.

I believe we have the capture of business process workflow variations down to a science. What we need next are the rules and data, not because we want to simulate or generate them at runtime, but because we need to know how many variations exist and how to test them. This is what optimal and measurable coverage is all about: having one test case (and no more) for each variation or set of rule outcomes.

In a structured and configurable environment like SAP, it is tempting to try and extract the rules from the application directly. This would have the advantage of being automated, of course, and thorough. The downside is that it may be both more and less than you need. It may be more than you need because the 80/20 rule applies equally to functionality, so you could be inundated with rules you don’t use.

It will also be less than you need, both in quality and quantity. Frankly it is illogical to test something by asking it what it does and then making sure it does that. That is not a test. A test asserts what it should do and verifies whether it does. Otherwise it is like asking the student to write the exam.

You will also get less than you need because some rules are tribal instead of configured. I am often struck by user ingenuity in grafting functionality onto an application. For example, users may adopt a reference only field such as comments and endow it with special meaning as a workaround for having a new field added. Or they may simply know from experience or actual company policy that certain customers receive specific treatment.

So I have come to believe that – unless you are blessed with current, complete and testable requirements - the best way to capture rules and data is to ask an expert to give them to you. That way you have independent verification. My experience, however, is that while experts are generally accommodating with input data values, they resist having to predict result values, which may require complex calculations or lookups. In an existing application the output values are often captured from the system itself, apparently on the theory that the system is working. The risk inherent in this habit, of course, is that the system might be in error, but the expert could be wrong as well.

The temptation then arises to write the rules so they can be executed and thus generate the values. Again, this has the ostensible benefit of being more efficient, but in fact it may cost more and be less effective. It will be costly because you are, in effect, rewriting the application functionality. This is a massive undertaking and every bit as likely to introduce error as the development of the application you are testing. It is less effective because it introduces ambiguity into the verification: is the application or the test calculation in error?

But however you end up with them, it all comes down to data values in the end. The fact is you have to have them to automate test execution. For now, most test automation systems – Worksoft Certify included – store data in tables and rows that are tied to variables and to business processes, but they are not tied to rules and outcomes. In other words, the relationships of the data values to each other and to the coverage matrix are not explicit.

And that’s the next frontier.

Enterprise Operations Assurance

2010-11-29T13:32:00.000-08:00

As technology becomes more integral to operations and interconnected across every aspect of the enterprise, the risk and cost of failure is expanding. Yet many companies introduce changes into their production environment on a frequent – even daily - basis, with only unit-level testing if any. Changes introduced through projects may be tested somewhat more thoroughly, but even their potential impact on connected systems is rarely tested at all.

The best explanation for this shocking but true predicament is the boiling frog syndrome. Biologists claim that if you drop a frog into boiling water it will jump out immediately, but if you place it in cold water and then heat it slowly, it will stay in until it boils to death. Apparently this phenomenon has to do with the rate of change and is often used to describe human tendencies to miss or dismiss significant changes that occur gradually.

Here’s how it happens to enterprise operations: A project is formed to develop, acquire or configure a set of functionality within a given timeframe. Almost inevitably the scope creeps up and surprises introduce delays, resulting in a threat to the delivery schedule. Unfortunately, delivery dates are seldom sacrificed due to other forces and expectations, including compensation incentives as well as operational and logistical momentum.

As a result, projects are often declared complete on the scheduled date, and any remaining functional gaps or issues are simply transferred into the maintenance organization, where they become part of the ongoing stream of changes and corrections that keep systems operational on a daily basis. This stream of changes is rarely routed through a test organization, instead relying on individual participants to verify their own work or perform spot checks as best they can. Enterprise level, end to end testing is entirely absent.

The net result of inadequate testing can, of course, be disastrous. Software failures have killed people, bankrupted companies, cost or lost hundreds of millions, and caused other losses and embarrassments too numerous to mention. This assumption of risk is tolerated either out of ignorance – as in the boiling frog - or faith that since nothing dramatic has failed in the past it won’t happen in the future. The fact is that not all errors are disastrous, and many organizations have become extremely adept at handling problems before they become critical.

But failures don’t have to be dramatic to be traumatic. Many companies find themselves struggling to manage their backlog as the maintenance budget drains away resources. Do you know how much of your maintenance costs are invested in improvements versus fixing problems? Does anyone know how much business productivity loss is caused by software issues? In other words, do you know what it costs not to test enough?

On the other hand, how is it even possible to test enough on a daily basis? Typical enterprises have hundreds of applications, interconnected internally or externally, across a complex platform landscape. Application ownership and business process expertise is typically organized within functional silos with little or no up and downstream visibility, and few or no resources dedicated full-time to testing.

What is missing in most companies is an enterprise level validation of end to end business processes, what I’m calling Enterprise Operations Assurance (EOA), that executes every single day before changes are accepted into production. The only way it is possible – and it is possible - is through automation. And the only way it is feasible is to focus on it.

Automate or Else

The reason automation is a must is because there simply aren’t enough hours in a day to manually execute all the end to end processes that operate the enterprise. For example, it took one company several days to manually execute all the way from Order to Cash; the automated test took 44 seconds. The majority of the difference was the logistics of coordinating individual roles and resources across functional silos, and the remainder was the difference in execution efficiency between a manual and automated test.

So what automation does is make it possible – on a daily basis - to execute the hundreds of end to end business process variations that are necessary to assure that essential enterprise operations will continue uninterrupted after any changes are made. These are the “no matter what” tests, as in “No matter what, we have to be able to sell our products, buy inventory, pay our people, etc.”.

Understand that EOA is not organized or prioritized by what is changing; changes are happening too fast everywhere and may not may not even be planned or documented. Instead, it is organized and prioritized around what must not fail, whether it has changed or not.

Focus or Fail

Although EOA won’t work without it, automation alone won’t bring it into existence. You still need to pay attention to it, and by that I mean someone has to take ownership of acquiring the business processes from all the silos, integrating them into a coherent, automated end to end sequence, and adding the infrastructure necessary to make it repeatable day after day or night after night.

The good news is, we’re not talking about adding a huge staff and workload. Business process experts already have to document their procedures for training and perform manual testing; all that is missing is the enterprise level view and the automation capability. Any additional investment will be more than paid for by the reduced costs of manual testing and the reduced costs of chasing problems due to unexpected impact from inadequate test coverage.

On the other hand, this is not a part time job or sideline. EOA must be a full-time effort on the part of dedicated resources. Furthermore, it cannot be optional. It must be executed each and every time before any changes are introduced to production, even if that is daily, nightly, or even more often. Because without focus and discipline, your critical enterprise operations remain at risk and it is only a matter of time before you will pay a steep price.

The bad news is that there is no clear owner. Because operations are distributed across functional silos, no single area has the vision or the mission to assure enterprise-wide operations. You might think IT should carry this banner, but most organizations are under unrelenting pressure to reduce costs, not increase them, and without a business imperative they are unlikely to invent work for themselves.

What’s the answer? I wish I knew.

Why Is Most Testing Still Manual?

2010-08-23T13:58:00.000-07:00

Despite decades of investment in test automation tools, techniques and time, the majority of enterprise application testing is still performed manually. I have always believed it is because the test tools were too technical and time-consuming to use, and if we could only make them easy and accessible then things would change. And while there may be some truth to this, it is not the whole story.

To understand the whole story, you have to step back and look at why testing happens, who does it and what this means to automation.

Why Test?

Testing happens because something is changing. In most enterprises, there are two streams of changes. One is the stream of projects, usually planned and executed within functional silos to advance specific goals. The other is maintenance or sustaining, which responds to production issues or sudden business demands. In this latter stream, changes may happen daily or even more often. These typically unplanned, out of window changes are a subject unto themselves for the next blog installment. For now let’s focus on planned projects.

When a project is planned, an allocation of time and resources is made to testing based on scope and risk. Of course even planned projects can result in both expected and unexpected outcomes, but the scope constraint imposed by the project necessarily focuses attention on verifying the expected results. Potential unexpected results are so undefined and broad in scope that they are either ignored or given cursory attention in the form of minimal regression testing. This means that most project tests are tightly focused around the specific changes being introduced.

This focus on what is changing makes sense from a resource and time allocation point of view, but for automation it’s bad news. The reason is that highly focused tests have limited reusability as a practical matter.

Here’s an example: Let’s say the enterprise has established a new distribution center that services a particular line of products and geographic area, and a project is launched to implement it and all its attendant business rules within the enterprise systems. As you would expect, the project calls for tests that verify that indeed the new center appears in the system and that its products can be shipped to the areas it services.

And while this all makes sense within the context of the project, it makes less sense to automate these tests for the simple reason that once the distribution center is operational, future projects will likely focus on something else and there will be no reason – and no resources – to keep running the same tests. Even if the tests are automated, why would they be re-executed and who would do it? Why would that particular distribution center be any more important than any other?

In this case, it’s not the ability or technology to automate that is missing – it’s the lack of interest in reusing the tests. And that lack of interest is only worsened by the way projects are organized.

Who Tests?

Projects are, by their nature, transient; they have a limited scope and duration. They are formed, executed, then concluded. The test resources assigned to enterprise application projects are typically business process experts borrowed for the duration from the operational area and functional silo that initiated the project. Thus, project test team members are also transients and scatter to other projects or back to their day jobs when it is concluded.

The upshot is that new project members may be unfamiliar with the artifacts and tools left behind by any previous project, or uninterested in them because of their irrelevance to the planned changes.

Unfortunately, this discontinuous approach to changes results in a low level of reuse, even when it might be justified. For example, I am familiar with a project that flatly refused to even explore, let alone use, an extensive library of automated tests created by a previous project team on the grounds that the previous project was concerned with a completely different set of changes and therefore the tests were irrelevant to them and not worth the effort of engaging the extra skill sets needed to take advantage of them.

Even documented manual tests may fall into disuse. Ironically, the more tests that are accumulated the less likely they are to be reused. New project members may not have the time or motivation to review hundreds or even thousands of previous tests to look for relevance; it is often easier and faster to just create something new than to locate and understand something old.

Who Cares?

In the end, manual testing predominates in this environment because there is little to no incentive – or budget or schedule – within most projects to invest in reusability. The project team is constrained by its own expertise and logistics to focus on the functionality at hand, with limited if any understanding of the potential impact on other areas or future projects.

Clearly there are some projects that are broad enough in scope to warrant automation, such as an initial roll-out or an upgrade that has wide-ranging potential impact. And just as clearly there are some aspects of almost every project that would justify reuse on the grounds that they are adding to the overall functional base and therefore represent a new component of risk. But the problem is that the transient nature and structure of the project itself does not lend itself to investing in reusability.

What is especially ironic about this conclusion is that in most enterprises, the only formal testing that ever happens is inside the project stream, yet the majority of changes actually come from the sustaining stream where testing is minimal and informal at best. But that’s a whole other subject for the next installment.

In the End, End-to-End is Everything

2010-05-17T11:26:00.000-07:00

Enterprises of any significant size depend on extensive software portfolios to operate. I know of several who manage more than two hundred applications and at least one whose inventory exceeds 500. Granted, some of these are small departmental systems, but many support mission critical operations. And, of course, most of these are undergoing continuous changes in the form of corrections, enhancements and updates. This results in a constantly shifting production landscape.

Yet software testing disciplines are almost exclusively focused on individual applications within the context of a given project. Traditional V-model approaches track the software development process for a particular system, for example, and while they may touch upon interfaces it is only for those systems directly connected. Even user acceptance testing is usually organized and performed within silos, with each participant focused on his or her area of expertise.

But the reality is that many of these systems are interconnected. Even if there is no direct interface, they may share data that originates elsewhere and terminates somewhere else, perhaps several applications removed. Thus, a change to one area often has unforeseen effects in another, and generally accepted test practices are not designed to uncover them.

The result is that even a well-tested application can wreak unexpected havoc once in production, and this is what keeps management up at night.

The solution, obviously, is testing end-to-end. That is, following information all the way from its origination to its termination, as in Order to Cash, Procure to Pay, and Hire to Retire. And even among these mega processes there are interrelationships: raw materials are procured that are processed by plant employees into finished goods for sale. Extensive integration is both the benefit and the curse of enterprise applications.

So why isn’t end-to-end testing widely practiced?

First, because it’s complicated. The fact is that enterprises are organized around functional silos and few, if any, truly understand all of the interconnections and relationships among them. Not everyone may even be aware of the existence of some systems, let alone how they affect, or are affected by, others. It takes a lot of diligence to identify the right end-to-end scenarios, then ferret out the right resources for each silo, then coordinate them all into a meaningful workflow.

Second, because it’s hard. Effective end-to-end testing requires a robust and controlled test environment that can reproduce – or at least simulate – production. This means installing the vast majority of the software portfolio and providing for both internal and external interfaces, either live or simulated. It also requires provisioning test data that is both comprehensive and coherent so that the flow of data from one end to the other is consistent. None of this is easy.

And, finally, it takes a lot of time. Just the logistics of coordinating each functional resource to perform their respective tasks in the right order is time-consuming. In one case it took five calendar days to execute one cycle of Order to Cash, most of it lost to the hand-off between people from role to role. This is often the showstopper, because once an application has completed its project and related testing, it is usually on a fast track to production. Or, even worse, the change is actually a fix to a problem that has created an emergency. But whatever the reason, changes may be made daily and typical end-to-end testing takes far longer. The result is that most end-to-end issues aren’t discovered until production.

But it doesn’t have to be that way.

End-to-end testing can be achieved and performed regularly, even daily. The key is to adopt the right strategy, implement the right tools, and enlist the right team.

Adopting the right strategy means including end-to-end testing as part of an overarching approach for every application. This means doing the analysis to understand the sources and uses of data for any application to reveal both immediate and far-flung interfaces. It also means applying risk analysis to understand which systems have the potential for critical operational impact, and making sure the test repository is kept current and updated as changes are made.

Implementing the right tools means deploying an automation solution that enables extensive end-to-end testing to run lights-out every night if necessary. In the above example of Order to Cash taking five days to execute manually, the same process was automated using Worksoft Certify…and it executes in less than a minute. How? No wait time, think time or hand-off time. Just rapid, error-free execution that is fully documented. This type of speed means hundreds of extensive end-to-end tests can be executed for each and every change before it reaches production.

And automation isn’t just for execution. Just knowing what changes have been made and what impact they may have on critical business processes is itself a challenging task. Wading through arcane release notes or change notifications or IT tickets is tedious enough, but even assuming you can understand what they say it doesn’t mean you can tell what the impact might be. Solutions like the Worksoft Transport Analyzer can simplify and accelerate the entire change impact analysis process so you know for sure what to test.

Enlisting the right team means not just finding and recruiting the right experts for each functional silo, but also assembling a centralized test team that can coordinate the human, hardware and software resources necessary to make end-to-end testing a reality.

Granted, even with these factors in place it won’t happen overnight. The key is to remember that even a little end-to-end testing is better than none, and all the time you will save from avoiding even one production crisis can be reinvested to continually expand your scope. With patience, effort and commitment you can eventually arrive at the point where your end-to-end execution validates hundreds of your most critical business processes...while you sleep, soundly.

Forget Regression Testing

2010-03-09T14:45:00.000-08:00

I must confess: I don't believe in regression testing. Not that I don't believe it can work, just that it doesn't.

After spending over 20 years in software testing and working with hundreds of IT shops, I have to report that effective regression testing--which is supposed to protect against unintended impact of intended changes - is about as common as development projects that are ahead of schedule and under budget. There are exceptions; some IT shops no doubt do a great job. But it's also true that some people have survived jumping off the Golden Gate bridge, which is not a strong recommendation for doing it.

So why doesn't regression testing work? I think it has to do with the way it is perceived and practiced.

For starters--and I'm only being halfway facetious here--what's with that name, anyway? Regression sounds bad--too much like repression, depression, and oppression. It sounds pejorative. And it doesn't seem to add a lot of value: Just makes sure that what used to work, still works. Most IT shops are rewarded for what they create--more features, new applications, cooler technologies. What glory is there in maintaining the status quo?

Furthermore, what's the difference between regression, system, or acceptance testing? Sure, there are subtle distinctions between the definitions of each, but at the end of the day, aren't they all about making sure the stuff works? Regression testing is about proving a negative: making sure that there is no unexpected impact from changes. How do you prove that something you don't expect to happen, didn't happen?

What's in a change?

Regression testing assumes you know what the application used to do before you make any changes, which implies a known set of requirements. But the fact is, few IT shops maintain current application requirements, relying instead on the individual expertise of testers conscripted from the ranks of business users. Thus, regression testing is only as complete as the knowledge of the person(s) performing the tests.

Aside from the inconsistency of this approach, there is a deeper problem. A "change" to an application's functionality is not necessarily the result of actual modifications to its code. In today's complex environment of interdependent and multi-tiered applications, there are myriad factors that can affect operations: a new version of the operating system, middleware, database, or change to the network topology or hardware configuration can result in downtime.

For example, I recently reviewed the production trouble tickets for a large IT shop and found that fewer than 25% of the incidents could truly be described as defects in the software itself. The rest were caused by ancillary system resources. Regression testers borrowed from the ranks of users can hardly be expected to know about, let alone understand and test for, the potential impact from changes of this nature. But if regression testing sounds bad and is poorly practiced, what's the answer? Well, how about renaming, redefining, and repositioning it?

What’s in a name?

First, lose the name “regression testing”. It not only conjures up negative images, it's also all about proving a negative, which is impossible.

Instead, start talking about Business Process Validation, which assures that critical business processes are still available and accurate after changes have been made.

Notice what we are going to assure: not what the system used to do, whatever that is, but that critical business processes are still available and accurate. The problem with regression testing is that what used to work is undefined. If you don't know everything the software used to do, how can you test it?

Consider the word critical, which means that BPV is concerned with those aspects of an application that are essential. In other words, we're not talking about every possible error condition, every bug ever detected, or each and every combination, boundary, data type, etc. This distinction is crucial because it implies risk management. If you know you don't have enough time or resources to get complete coverage, then prioritization is key.

Now think about a business process. The "business" half of this term places this type of testing squarely outside of development. The "process" half says that these are user scenarios, examples of everyday tasks that run the business, not mathematically derived test cases. Taken together, this means that the business-user community is both a participant and a beneficiary in BPV.

Finally, let's examine the phrase “after changes have been made”. Notice the word "changes" is not qualified: Instead of just modifications to application software, a change can be to the hardware or any aspect of the environment--even the business process itself. Critical applications reside in highly complex, interconnected environments, relying on multiple tiers and layers of functionality. It is unrealistic and downright naive to apply application-centric test techniques to integrated IT environments.

The implication is that the test environment should be a microcosm of your production environment--not just in its configuration, but in its processing cycles. Thus, BPV testing can't be cadged from a corner of a development system, making do with volatile data and unstable software configurations. It requires a tightly controlled and well-managed test environment.

The last detail to settle here is exactly where Business Process Validation fits in your delivery process. The most logical place is as a condition prerequisite to promotion into production. In other words, BPV should be the gateway to operations, the point at which the business confirms that it can continue to carry on uninterrupted. It happens last.

Why is that so important? Because someone in a deadline crunch might slight a process as obscure as regression testing, but who would dare fail to assure that critical business processes are available and accurate? Perish the thought!

Rewriting The Rules

2009-11-16T12:44:00.000-08:00

We’ve all heard the rule of software delivery – you can choose any two of fast, good or cheap, but not all three. If you want it fast, it either won’t be good or it won’t be cheap. Or, conversely, if you want it to be good then it either won’t be fast or it won’t be cheap, and so forth. You get the idea.

Until now, that is. The engineering team at Worksoft has managed to deliver excellent quality both quickly and economically. They did it by “eating their own dog food”, or using Worksoft Certify to test itself, then wiring comprehensive test coverage around an agile model.

Here’s how it works. First, the fast part: the team has adopted an agile approach to development that calls for continuous integration. Each developer is responsible for introducing changes, testing them, and integrating them into the overall build that is then also tested. This means a fully tested build is available on a daily basis.

Next, the good part: the QA team invested the effort to develop an automated baseline regression test of core Certify functionality. This suite, containing hundreds of tests, is used to test the daily build. Because of automation it takes less than two hours to run, so each developer can execute a complete regression suite before he or she introduces the change to the build, with enough time left over to run the same tests against the full build.

Then the team took it one step further. Not only does each developer submit changes, they also submit new tests to exercise the functionality they added or changed. Virtual machines are set up with Certify pre-installed and pointed to a database (the upstream database) of functional tests that are continuously augmented by the developers. Integration builds are triggered by software check-in or through a timed nightly build. On successful build completion, the Certify virtual machines pull down the new version of Certify and run the functional tests (this process is very fast since tests are automated and does not add significant overhead to the build process). Results of the test run are then emailed to the build team for review.

So while Certify gains capabilities on a constant, continuous basis, the test suite also keeps pace. Hundreds of tests are on their way to thousands, and because of automation the entire suite can still be executed in a couple of hours while the developer moves on to the next task.

Finally, the cheap part: by producing a quality product literally every day, Shoeb’s team can focus their efforts on creating new value instead of fixing what broke in the last release or testing their changes for the next one. The developers are much more productive and so are Worksoft’s customers, since neither is distracted by chasing the problems that should have been caught in test. The net result is a better product with less time and resources.

Aside from redefining the rules about what is possible, by becoming dedicated users of the very product they are developing the engineering team has a much better grasp of customer needs and perspective. In fact they often identify new requirements before the customer does. In essence, the engineering team runs Certify in a production environment. They install and use Certify before any other customer, so any issues that managed to escape the comprehensive regression test are usually caught the next day.

And the best part? Successful, satisfied customers.

The Logic Against Using Logic

2009-09-01T11:23:00.000-07:00

There is no doubt that you can reduce the size and number of test cases by focusing on reusability, the question is whether that should be your priority. My experience says no.

Here is the problem: If there is a process whose steps vary based on rules, reusability leads you to incorporate the rules into the test as logic to decide which steps to execute in what circumstance. That way you can reuse the steps, right?

The trap is that now you are developing a set of rules that must mirror the application logic, so not only do you have to maintain changes to the steps, you must maintain changes to the rules. Nor can you document the test behavior in advance: It can only be known at runtime when the conditions are satisfied. Your test steps, while fewer, will be more complex, fragmented and difficult to understand and therefore tricky to maintain.

Reusing test steps as a precious commodity may make sense if it takes a costly programmer to develop them, but with a graphical data model like Certify it doesn’t. Copying, building or modifying steps– and especially maintaining them - is fast and easy enough that you don’t have to be afraid of a few extra. You trade redundancy for readability and predictability

And the fact is that the very definition of a test case is definitive: that is, the test must specify what is expected. Adding a bunch of “if, then and else” clauses introduces inherent unpredictability and introduces the possibility of the very logic errors that the application developers are subject to. When you don’t know what to expect, I maintain, it is no longer a test - it is an experiment.

Now, does that mean you will never have any logic? No.

The logic you still need is the intelligence to enable unattended test execution, so that if something happens that is unpredictable or unexpected, the test can compensate and perhaps continue. There are three basic types of execution risk:

Known risks are those you know exist and can remove through logic or other means. For example, if you can’t control or predict the test environment data - a different problem altogether - you may want to verify inventory to assure quantities are sufficient before you create a sales order. You know what the risk is and you remove it. Risks of this type, however, should really be solved at the root. For example, establish control and predictability of your test data so there would be no question.

Known unknown risks are those you know exist but can’t remove. Timing is a good example of this. You cannot predict nor control user response time or system execution time and so need logic to adjust to actual behavior. For example, waiting for the system to acknowledge that a transaction has posted so the test can proceed will likely require some logic to detect and wait for the response.

Unknown risks are the surprises – risks you don’t know exist but are possible. Errors can spring from beneath a number of rocks including the entire technology stack, data sources and of course the application under test. There is no way to predict every possible error from every player in the operating environment. For these risks, you need global recovery logic that returns the context of the test to a known place from which the next test may proceed. Without this basic logic, your first unexpected response means the end of the entire test session.

Now, you might hear “logic” and think programming. Not necessarily. In Certify, for example, complex decisions can be made by simply selecting from available options in a list. But just because it’s easy doesn’t mean it’s always a good idea. The less logic you write the better. So use it sparingly, only when you can’t remove the risk through other means. But when you do use it, try to make it as reusable and straightforward as you can. For example, create shared global recovery steps.

There are many other benefits to designing tests without logic. They can be used for documentation, training and compliance. Their readability makes them easier to understand, debug and maintain. Yes, maintain. Even though you may have more tests and steps overall, in a data model like Certify it is easy to locate steps that need changes and in most cases do global maintenance.

The key is to remember that test design should always focus on the expected and the unexpected should be minimized with proper test environment planning, setup and control. In the end, the best logic against using logic is that predictability trumps reusability.

The Secret to Automation Success

2009-06-11T13:31:00.000-07:00

If you are among the thousands who are struggling with a test automation record/play/script tool, I can tell you the secret to succeeding – or at least giving yourself the best possible chance to succeed – in a single sentence:

What you have is a tool and what you need is an application.

What’s the difference? Let’s use an analogy. Imagine that my accountant is keeping the books in a spreadsheet and, naturally, it is taking too long and he is making too many mistakes. So, I decide that automation is the answer. Would I record the current manual process and try to replay it every month? Would I buy him a compiler and expect him to develop an accounting system on the side, while he continues to keep the books by hand? Would I hire programmers to develop an accounting system for him?

I’m sure you are thinking – you should be, anyway – that all of these options are comical. The obvious thing to do is to buy an accounting application.

Now just substitute “tester” for “accountant” and “testing” for “accounting”, and you get the picture of why test automation has failed. Companies think they are buying a test automation application when what they are buying is a tool that lets them choose one of the above three options. They have no idea what they are getting into.

Recording a manual task does not make it automated. It is simply a stream of events at a given time without structure, logic or documentation. We test software because it changes, and maintenance of these sessions is difficult to impossible. Capture/playback is a marketing technique, not an implementation one.

Expecting manual testers to design and develop their own application while they are also required to meet schedules for the application under test is also absurd. And the best testers usually aren’t programmers anyway, so you need additional resources and skills to even attempt it.

But while buying a test automation application is an obvious answer, testing is not as easy as accounting. Accounting rules are standardized, testing isn’t. There is a never-ending mix of technologies, architectures, languages, and functionality that makes it impossible for one test automation application to handle all possibilities.

Or does it?

What made SAP so successful was the idea that functionality can be represented as a data model instead of as code. Instead of each customer developing their own custom applications because their processes were unique to them, SAP allows them to buy a packaged application and then customize it…without writing any code except perhaps in fringe cases.

And therein lies the core difference between a tool and an application: code versus data.

With a test recording and scripting tool, code is accumulated. Whether it is recorded, generated or developed, it must be managed and maintained. The problem is that application functionality expands over time, but the time and resources dedicated to testing are either flat or declining. This means more and more test code must be created and – especially - maintained with the same or less time and resources.

The math just doesn’t work.

So what’s the answer? A test automation application that uses a data model instead of code.

Now, you should be saying to yourself “Well of course she thinks that’s the answer. She’s a vendor and has a product she wants to sell, so her opinion is suspect.” And you would be partly right.

You would be right in that I am a vendor and do have a product to sell. I founded a company that markets a product that does exactly what I am describing. But I have also taught training classes, tutorials and seminars on how to build one yourself, and have had many successful students who were able to use the tools they had at hand to make it work.

But you would be wrong that my opinion should be suspect as a result. In fact, you should be even more skeptical if I didn’t have such a product, because it would mean I was one of those people who teaches about things I’m not capable of doing. If I hadn’t developed it, how could I be so certain that it works? What value would my opinion have if I had not put it to the test?

The bottom line is not whether or not you buy my product, it’s whether you buy what I’m saying

If you do, then your options are to do it yourself, buy it from Worksoft, or buy it from someone else. And if you decide to do it yourself because you already have an investment in a tool or because your application is not a good fit, I’ll do all I can to help you. Your choice.

Automation Works When You Do

2009-05-24T14:46:00.000-07:00

Why is the majority of testing still performed manually? True, many tools are too hard to use and maintain, but is that the only reason?

No. No matter how good the technology is, it is still only a tool. It won’t jump out of the box and start testing your software by itself. My favorite analogy is that buying a test automation tool is like joining a health club: the only weight you have lost is in your wallet. You actually have to go to the club- consistently – and use the proper equipment the right way to see any results. You also have to set goals so that you can develop an exercise plan.

For automation, you have to invest the effort to design and build a reusable library of business process tests to reap the benefits. Since you can’t automate something that is not defined, you have to focus on discovering and documenting the current state of your business processes first. For some lucky companies, their business process documentation is current and complete, but not so for the majority. And you can’t skip the risk assessment, either, because I guarantee you have more functionality than you can cover – at least in the near term – and you need to target your efforts.

You can’t just hire someone to do it for you, either. Don’t get me wrong – having consultants help you get started down the right path is invaluable – but it doesn’t mean you abdicate all responsibility. In keeping with the fitness theme, hiring consultants to implement test automation is like hiring a personal trainer. It doesn’t do you any good if the trainer works out instead of you. A trainer can show you the proper form by working out with you, but you have to do your part.

Like exercise, automation is all good, but follow the same advice for automation as you do for exercise: start slow. I don’t mean you should deliberately waste time, but be willing to accept gradual progress. Tackle your highest risk processes first, and once you get those automated then move to the next most critical, and so forth. Don’t set an unrealistic goal of total coverage in your first cycle or you will delay the benefits that even a little more coverage will bring you.

And finally – but most importantly - there is the lifestyle change. To stay fit, you have to incorporate both diet and exercise into your daily life…forever. The good news about exercise is that the more muscle mass you develop the more calories you burn even at rest. So, the more you exercise the faster you see results. The bad news is that your metabolism slows as you age, and if you slack off or return to prior bad habits, the effects wear off and you will be back where you started or worse.

It’s the same with automation. You have to integrate automation throughout the complete life cycle, and continually assess your risks and improve your coverage every time there is a change. You not only have to keep your existing tests current, you have to keep adding new ones, because the longer your application is in use the more functionality it will have and the greater the risk of unexpected impact. So you can’t just plan a quick project to automate and then stop: those tests will eventually become obsolete and your application will continue to expand.

But just as more muscle mass burns more calories, an expanded test library yields better and better returns. All the time you save by liberating your manual resources is returned to you in the form of improved business processes, more enhancements and streamlined operations. And the greater your test coverage the fewer emergencies and outages you will experience, which will improve user productivity and free IT resources to reduce the backlog and improve response time to new business needs.

So if you have made the investment in test automation technology, and perhaps even hired consultants to get you jump started, don’t stop there: integrate automation into your day to day lifecycle so that your benefits will get better and better. And here’s to your health.

The Hidden Costs of Testing

2009-03-16T10:48:00.000-07:00

Even though testing consumes an average of 50% of IT projects, I’m willing to bet there is not a line item in most corporate budgets called “testing”. While those that develop software internally may have a QA department that is budgeted for, those that license packaged applications probably don’t because test resources are borrowed from the business or other tasks. As a result, the cost of testing – which is substantial – is effectively hidden.

Even worse, I am certain there are no companies whose financial statements have a line item called “inadequate testing” or “software failures”. Yet the evidence is irrefutable that the cost of failures in production can be significant or even catastrophic. There are direct costs such as increased support or helpdesk demands, rollbacks, re-testing and emergency transports, but also indirect costs in the form of lower productivity or even reduced revenue or lost customers. Then there are the ripple effects – resources that are diverted to handle emergencies are not available to complete planned projects, resulting in an overall slowdown.

It’s not that these costs are absent, they are just hiding in plain sight as increased operating costs, reduced revenue or market share, and overall decreases in productivity and profitability.

So why does this matter? For the simple reason that if you can’t measure it you can’t manage it, and without an accurate accounting of costs management can’t make informed decisions about the drivers that affect testing and all the related effects of trade-offs.

Maybe it’s my inner accountant that bristles at the idea of unacknowledged expenses, or maybe it’s the recent hoopla about how transparent the federal budget is or isn’t, but no matter what the reason I think it’s just plain wrong to essentially ignore such a critical business driver.

The Cost of Widgets

If a company is making widgets, the generally accepted accounting principles (GAAP) that must be satisfied for a clean financial audit demand a detailed cost accounting of all the components that create the widgets: direct variable costs such as materials and labor must be tracked and indirect fixed costs such as supplies and overhead must be allocated. And this is rightfully so: otherwise, how can the company determine if it can make widgets at a profit, or at what production level it must operate to breakeven?

This approach to accounting for costs enables informed decisions. For example, the decision to invest in automation equipment on an assembly line can be objectively evaluated by comparing the costs of performing a particular task manually versus automated, then calculating how many widgets over how much time would be required to recoup the cost. Throw in the cost of capital, estimate a useful life for the equipment, and viola: you have a dispassionate financial analysis of the return on such an investment.

The Cost of Software

When a company makes or buys software, these rules are strangely absent. The only GAAP rules center around whether the costs should be expensed or capitalized, based on the useful life of the application. But the detailed tracking of cost components is missing. At best, companies might track the project timeline by task, but few if any can report with any accuracy how much it costs them to analyze, design, develop, test, implement and maintain their applications.

This lack of accounting for software costs disables informed decisions. For example, I asked the CIO of a global company how much was spent on testing during their last upgrade. He estimated “A lot…at least 20%.” The actual cost, according to the project manager, was closer to 70%. This level of misinformation makes it impossible to adequately evaluate decisions such as the automation of testing. It is especially frustrating when executives ask for a ROI analysis for test automation yet can’t supply the most basic of metrics about their existing cost structure.

The Cost of Ignorance

In this case, ignorance is not bliss. It leads to poor decisions that result in higher costs, lower revenues and decreased profitability. There are even cases – some of them famous – where inadequate testing literally bankrupted the company. At a minimum, the hidden costs of testing mask forces that affect key business drivers and cripple effective management of scarce human and financial capital.

So how good is your software cost accounting? Take this quick test to find out:

· Of the total cost of your last project, how much was allocated to each task of the application life cycle?
· Do you account for the time spent by business users and other borrowed resources in doing testing? What was the opportunity cost of other tasks they could or should have been doing?
· How many emergency transports did your company perform last year, at what cost? What was the cost of delaying planned projects as a result of these unplanned activities?
· What was the impact on revenue and/or costs of software errors or lack of availability?

If you can’t answer these questions, the cost – and value – of testing is hidden from view, and as long as that is true you won’t be able to make the best decisions about an activity that consumes as much as half of your IT projects. And by any measure, that’s significant.

The Cost of Doing Nothing

You might agree in principle that better information would be helpful, but since you’ve managed all these years without it, what’s the rush? Why do anything different, especially now? Precisely because this is the best time to get smart about costs. When the economy is unpredictable – some might say unprecedented – it is time to change your thinking.

As Yvonne Genovese, vice president distinguished analyst with Gartner Research, points out: “We find that automating testing has a very compelling argument in that it’s important during the downturn to help us get more efficient – and we are in dire need of ways to cut costs in our current IT environments,” she said, “Manual testing is very inadequate, expensive, and introduces risk in the environment, where automated testing will help us take advantage of uncertainty and position us for the future because it allows us to test –and validate--across the end-to-end of the process, capture the test for reuse, and document the process, all while freeing up valuable resources.”

Exposing the hidden costs of testing may reveal opportunities for advantages that will pay benefits far into the future. There has never been a better time.

Test Data: The Hard of the Matter

2009-02-11T13:39:00.000-08:00

Here’s a tough truth: If you can’t get control of your test data, you can forget test automation. The core value proposition of automation rests on reusability, and if you can’t reuse the data you can’t reuse the tests. Even manual testing requires test data, of course, but a manual tester can try to find or create the data they need on the fly.

My experience shows that testers spend 80% or their time or more just trying to locate data they can use or entering the conditions they need. For more on this, see "Automated or Not, It’s All About the Data" (http://www.stickyminds.com/s.asp?F=S9033_COL_2).That means having reusable data even for manual testing would improve productivity by orders of magnitude.

So what’s the big deal? Can’t you just copy production data? Aren’t there tools out there that let you extract snapshots from databases, scramble existing data or generate fake values?

The bad news is, these traditional techniques usually don’t work. The good news is the solution may be easier than you think.

The Problem with Production

Most companies use production data in some form for their testing. For one thing, it’s realistic – after all, it’s from production. For another, it’s easy – it’s data you already have. But there are problems.

The first problem is that it is probably so much data that it’s costly to copy and store. Second, all that volume makes it hard to find the exact conditions you need for a particular test scenario, and it may not even exist in the form you need. Third, it’s dynamic – a constantly moving picture that is too unpredictable to yield any reuse.

And lately there is a fourth reason: privacy. It’s highly likely that some of the data is confidential and cannot be legally disclosed to others, including testers. There are the obvious fields like social security numbers, but others may include personal data like addresses, account numbers, and any transactions related to financial or health matters. See "Keeping Secrets: How Data Privacy Affects Testing", http://www.stickyminds.com/s.asp?F=S8327_COL_2 for more detail on this issue.

And yes, there are tools that can help with these problems. But it’s not that easy.

The Trick with Tools

Database tools have been around for a long time and are widely available. Most of them were developed to test databases by generating high volumes of data or extracting selected subsets, but some are specifically targeted at testing and can locate, obfuscate or create required data values.

The trick is that it’s tricky to selectively extract or create a meaningful, coherent set of data. By meaningful I mean that it contains the test conditions you are interested in, and by coherent I mean that all of the related data is included. It is not as easy as taking every Nth record or some flat percentage of the data: complex interrelationships must be maintained between data elements.

For example, testing customer orders might require the customer master record, any related contracts on file, all transactions for that customer, plus all of the warehouse locations, product inventories, shipping codes, commission records, and myriad other data elements that touch the customer or the transactions - or anything they touch.

Furthermore, few applications operate on a single source of data. Many have interfaces to other applications that take the form of even more files in a wide array of formats. Some of these interfaces are real-time, some are batch, but all must be coordinated. While database tools may help you trace the relationships between tables and fields, they may break down when external files and formats are in the mix.

And finally there is the question of dates. Dates abound in most applications and are often central to calculations and event triggers. The date of an order may affect its pricing based on contractual terms. Posting a payment in one period versus another may create a late fee or interest charge. The dates of shipments or receipt of goods may trigger automated inventory orders, and so on and on. Database tools may know about table relationships but they don’t know about date dependencies.

My experience shows that despite their availability and relatively sophisticated capabilities, few of these tools are actually successfully deployed for testing because the effort and skills required to make them work is too high.

The Advantage of Automation

So what does work? Ironically, test automation presents both the challenge and the solution. It is the challenge because, as pointed out, automation won’t work without reusable test data. It is also the solution, though, because automation can create the data it needs.

Think about it. If you need a customer account with particular conditions for testing, you can either try to find it or you can create it. Finding it may take a lot of time and it may not even exist in the form you need. The advantage of creating it is that you know it exists and that has the right conditions because you put it there. Another plus is that the very act of creating the customer is, in itself, a test.

Manually creating all that data is not practical, but with automation it makes sense. Test automation tools can type in data quickly and accurately. By simply planning our your test cycle carefully, you can be sure that all the data you need is there how and when you need it…and expand your test coverage along the way. See "The Test Automation Timetable: Altered States" http://itmanagement.earthweb.com/entdev/article.php/622301 for more on automating data states.

Now this doesn’t mean you won’t need a starting point that includes basic master data. Believe it or not, trying to start with a blank database is almost impossible: there are far too many pointers, stored procedures, and other arcane contents that you could not understand, let alone create, in your lifetime. So you will still find yourself starting with some production or sandbox data just as a backdrop, but you won’t use most of it.

Instead, you will use the data you design. And that’s another benefit of automated data: You can design exactly the conditions you need, and because you are in control you can be sure that dates have the right relationship to each other and to the system date.

You will still need a strategy for interfaces that will probably include maintaining copies of files and massaging the values, but since you are in control of the core database contents it will be easier to know what data you need.

If this sounds too simplistic, I can tell you that some of the largest companies in the world, with massive, complex IT landscapes that span the globe, have successfully automated both their testing and their data using this approach.

Do the Math: Automation is No Longer Optional

2009-01-07T11:28:00.001-08:00

I don’t get it. Software development has evolved from a slow, painstaking, manual discipline to a rapid, iterative, highly automated process. Computers have escaped glass-walled rooms with raised floors to land on palm-sized devices. Applications have shifted from back-office productivity aids to front line competitive weapons.

Yet the majority of functional testing is still done manually.

It’s not for lack of technology – automation tools have been around for decades. I should know, I co-founded a record/play/script company in the 1980’s called AutoTester and followed up with Worksoft to take the coding out of automation. And of course the test tool ecosystem has expanded exponentially with a broad range of companies and products.

Neither it is a lack of need. The graph below is my favorite because it says it all: over time, functionality grows but testing resources don’t. Unless you can employ automation to create a cumulative library of tests, you are doomed to steadily decrease coverage and increase risk with each succeeding release.

The risk is also exploding. Whereas before software failure likely meant that the accounting staff had to revert to calculators for a day or two, today errors can instantaneously affect thousands or even millions of customers, tainting the company’s brand, reducing revenue and driving up costs. In some infamous cases, inadequately tested software has cost hundreds of millions, even billions, of dollars.

So what gives?

As far as I can tell, it’s a combination of four factors:

1. An undefined test process. You can’t automate what isn’t defined, and manual tests are by their nature informal, usually relying on the knowledge and skill of the individual tester. If you don’t have a handle on what you need to test, you can’t predict the time and effort for automation. See “The Importance of Testing Software Requirements” http://searchsoftwarequality.techtarget.com/news/column/0,294698,sid92_gci1275907,00.html for a deeper dive into this topic.

2. Lack of software testability. Many modern application interfaces are dynamically generated on the fly, resulting in window and objects that have no useful or persistent names. A lack of testability standards may also cripple automation by presenting components that tools can’t interact with. These combine to make automation either impossible or so time-consuming that the payback isn’t there. See “Automated Testability Tips” http://www.stickyminds.com/s.asp?F=S6013_COL_2 for more.

3. Inadequate resources. Too many companies think that all they need to do is write a check to the tool vendor and automation is on the way. But buying a tool is like joining a health club: you still have to invest the effort. And you can’t expect existing staff to keep up with manual testing while automating in their spare time. See “Join the Club” http://www.stickyminds.com/s.asp?F=S10695_COL_2

4. Unrealistic expectations. Record and play demos paint a false picture of what automation is all about, so management believes that it requires nothing more than capturing the manual process and therefore expects immediate results. This leads to a failure to instill development disciplines and allocate the proper time, attention and resources to getting the process defined and automated. See “The Demise of Record Play Script” http://www.worksoft.com/pdf/TheDemiseofRecordScriptPlay.pdf for a better understanding of the true costs of record and play.

So what to do?

First of all, face up to the fact that manual testing is no longer a viable option. Don’t get me wrong – you will never automate 100% of your testing, simply because some tests are not worth automating and others, like usability tests, depend on actual user interaction. But the vast majority of functional testing can, should – even must – be automated in order to achieve anything close to comprehensive coverage on time and within budget.

Second, develop and present a reasonable plan for automation that takes into account the current state of your test process and sets realistic expectations. Accept a strategy for gradual progress, and reject any plan that fails to account for the additional time and effort that will be required to get established. Granted, automation saves time and resources, but that’s only after you get it into place.

Third, don’t expect to exclude your experts. Unless you have clear, current and comprehensive test case documentation (and who does?), you will need access to your best and brightest to help you identify the most critical aspects of the application and the business rules that govern them.

And fourth, attack the test data and environment issue head-on. Simply put, automation requires repeatable, stable data, and this is one of the most challenging components of any automation strategy. In fact, it’s such an important topic that it deserves its own separate treatment.

So stay tuned.

Forget Quality, Focus on Value

2008-11-05T12:15:00.000-08:00

In my last blog entry I said that it is not a tester’s job to find bugs. Now I’ll take it one step further – testing isn’t “quality assurance” either. I have never liked this term to describe testing, because testing can’t assure quality. You can’t test quality into a product; it’s either there or it isn’t.

But there is another reason, too. My observation is that companies aren’t looking for quality in their software products, they want value. So what’s the difference?

Let’s use an analogy. You might want a high quality car, but that doesn’t mean you are willing to pay $250,000 for a Rolls Royce. There is a limit on what your transportation is worth to you, especially if you could buy a house for the same price. Neither would you be willing to walk or take the bus while you saved up the money to buy a Rolls.

Instead, you would buy a car that you could afford that still met your needs for transportation and safety. For some people that might be a used Honda, for others a new Lexus, and for some a Bentley might make sense. It all depends on your perception of the value of transportation and the trade-off in time and money you have to make to get it.

Software is no different. Companies buy or build software because it provides functionality they need. While they might say they want zero defects, they are rarely willing to pay a lot more or wait a long time to get it. Instead, they will balance the utility of having the software against the inconvenience of any defects it might have. They will reject the software only if those defects prevent their enjoyment of the functionality in some material way.

The proof of this is everywhere. Companies ship or install software with known defects all the time. Most software companies have backlogs of reported defects that number in the hundreds or even thousands, and some of them have been there for a long time. So why don’t they get fixed? Because to fix a defect you have to trade-off against adding a new feature or delaying a release. Quite simply, it’s a value judgment.

So why does this distinction matter?

It matters because it causes a disconnect between the testers and the users. Testers report their results as a list of defects, usually categorized by severity measured by system impact. For example, a severity 1 might cause a crash or loss of data, severity 2 means a feature doesn’t work, and so forth. I have been in too many release meetings where the status is reported as “X number of sev 1, sev 2 and sev 3” issues while the users’ eyes glaze over. What ensues is a negotiation over downgrading the severity of an issue so that some arbitrary policy – “No sev 1 issues!” - can be mollified.

At the end, the testers come away resentful that their hard work in uncovering these issues has been devalued or ignored. Worse, they fear – sometimes rightly – that they will be blamed for the poor quality once it goes into production. Users, on the other hand, become frustrated that testers seem to find never-ending reasons to delay or deny their enjoyment of the software.

What to do?

The easiest place to start is by changing the conversation to be value-oriented. Instead of reporting “We found a high severity defect” or even “The product drop-down list is corrupt”, say “You will not be able to enter an order”. This shifts the perspective from quality to value. You are no longer talking about the impact on the system, you are talking about the impact on the user. Then let the users evaluate the risk and the priority.

Then, try not to prejudge what is and isn’t critical to the users. You might think that not being able to enter an order would be unthinkable, but the users might decide that the majority of orders are transmitted electronically anyway, and waiting to fix this will delay the availability of a new product or service that generates more revenue than the handful of manually entered orders.

Next, check your virgin sense of quality at the door. Don’t imagine that you are there to find bugs or “assure quality”, realize that you are there to deliver value, and value is whatever the users say it is. Your job is to help them understand the trade-offs between enjoying the functionality and dealing with the defects so they can make an informed decision.

Finally, make sure the users understand where you stand. Ask them what is most important to them about the software and test that first. Don’t let them get away with hazy or missing business requirements; insist that you know the critical processes the software must support and make sure they work. Treat users like both a customer and a partner: a customer because you work for them, and a partner because you must work together to get anything done.

And let’s not forget to lose “QA”. How about something that creates value…like Business Process Assurance? See “A New Twist on Test Processes” at http://www.stickyminds.com/s.asp?F=S7069_COL_2. Also, “To Win, Change the Game” at http://www.developer.com/java/article.php/614401.

Test Smarter: The Top 3 Testing Myths

2008-08-26T09:21:00.001-07:00

Welcome to my blog. My hope is to share 20+ years of testing experience gained the hard way to make it easier for you. The best way to start is to dispel the basic myths that have been around the industry ever since I got started. From there, we can move on to more positive subjects, such as how to really succeed as a tester by testing smarter. Along the way I’ll also share links to other columns and articles by myself and others to expand and expound on the topic at hand.

So, here are the top 3 myths we need to lose so we can move on to what testing is really all about:

Myth #1: Your job as a tester is to find bugs. This premise is fundamentally flawed because it requires you to prove a negative, which is that there are no bugs remaining when you are finished. Sorry, but that is neither possible nor practical.

It is not possible because the potential combinations of user behavior, system environment and code conditions are essentially infinite, so you will never be finished. It is not practical because it would take too long and cost too much to even try. By the time you completed the exhaustive testing necessary to even approach full coverage the software would be obsolete and your users would have found another solution. Perfect tax software is worthless on April 16th.

Further, this myth leads to the belief that the more bugs you find, the better. This in turn encourages testers to engage in bizarre behavior and to focus on the fringes of functionality in the hopes of finding more bugs. Anyone can get software to fail if they try hard enough, but frankly users are more interested in what works than what doesn’t. They developed or bought the software because they need to use it, and finding random reasons why they can’t will not make you popular.

Reporting weird bugs invites developer derision and user frustration because it diverts time and resources from the real goal: making sure the software does what the users need. If you don’t believe users will sacrifice quality for functionality, look no further than the 50,000+ known bugs shipped with Windows itself.

This myth also discourages testers from exercising the core functionality – the so-called “happy path” – that is most widely used because it is less likely to yield bugs. But just because a feature has worked every previous time doesn’t mean it can’t fail this time, and if it is vital to the user then a single failure is one too many. Waving the list of bugs you found in the weeds will do nothing to placate users if mainstream functionality fails. For more on this, see “When Bugs Don’t Bug Me” at http://www.itmanagement.earthweb.com/entdev/article.php/621801.

So what is your job as a tester? It’s actually much more fun and interesting than just finding problems…but you’ll have to stay tuned to get the rest of the story.

Myth #2: Test new functionality first. This one has a certain surface appeal, since of course new functionality is unproven. The issue is that if it is new then users aren’t using it yet, but if it is old then the odds are they are. A broken new feature may be embarrassing, but breaking functionality the enterprise already relies on could be catastrophic.

This myth leads to the dangerous practice of focusing on changes first and only doing regression testing if there is time left over. The fact is that testing must be risk-based, and change is only one risk factor. Others include volume, financial impact, and exposure. Clearly a new feature may be high risk for many reasons, but the fact that it is new is only one of them.

And, since the overwhelming amount of testing is still performed manually (more on that next), this inevitably means that regression testing is slighted. There just isn’t time to test everything that is changing plus everything that isn’t supposed to change.

The irony is that over its lifetime, software tends to become less reliable, not more. This is caused by a variety of factors: developer turnover, repeated patches, constant enhancement, and changing technology landscapes. The volume of code and the functionality it supports continues to expand while developer knowledge is lost over time and turnover, and as a result the software becomes increasingly more complex and less stable. For more on this, see “Anatomy of Ugly: How Good Code Goes Bad” at http://www.computerworld.com/printthis/2005/0,4814,106105,00.html.

Thus, each successive change increases the probability of unintended consequences. Coupled with decreasing regression test coverage, this is a formula for disaster. Happily, changing your strategy can make the most of your efforts and win you popularity with developers and users alike.

Myth #3: You can only automate regression tests.The overwhelming majority of testing is still performed manually, even though test automation tools have been around for decades. You have to wonder why. Part of this is because most testing is centered on new functionality, as previously discussed, and the rest is because of code-based test automation tools.

Originally, test automation tools were based on an approach called record and play, or capture/playback, which required that testers perform the test manually as it was recorded into a script file. This meant, of course, that you could not create the test until you could perform it, and you could not perform it until the software was available and working as expected.

Later the scripting languages became more sophisticated and theoretically you could code tests in advance. Unfortunately the effort to develop and maintain the code was so onerous that it didn’t make sense to do it for any functionality that was in a state of flux. Thus, testers still needed to wait for the application to stabilize, and even then they only automated regression tests for features that weren’t likely to change. When they did change, it was usually easier just to start over than to modify the code

What makes this whole idea silly is that we test software because of changes. Even so-called regression tests can be impacted by new capabilities. By adopting an automation approach that is so sensitive to change that affected tests are easier to throw away than fix, you lose the primary benefit of automation, which is reusability that enables cumulative coverage. And, if you lose this benefit, then the whole value proposition of test automation is also lost: you spend so much developing and re-developing tests, or attempting to maintain them, that you don’t save enough time and money to pay for it. Thus, shelfware is rampant for test tools. For more on this see “The Demise of Record/Play/Script” at http://www.stickyminds.com/s.asp?F=S10939_MAGAZINE_2.

The good news is that test automation has evolved away from code to a data model that not only allows tests to be easily and quickly developed before the code is available, but also supports automated maintenance that makes reusability a reality.

So now we have cleared away the misconceptions and set the stage for an exploration of how to test smarter, giving you the opportunity to be successful and enjoy yourself along the way.

See you next time.