The Logic Against Using Logic

There is no doubt that you can reduce the size and number of test cases by focusing on reusability, the question is whether that should be your priority. My experience says no.

Here is the problem: If there is a process whose steps vary based on rules, reusability leads you to incorporate the rules into the test as logic to decide which steps to execute in what circumstance. That way you can reuse the steps, right?

The trap is that now you are developing a set of rules that must mirror the application logic, so not only do you have to maintain changes to the steps, you must maintain changes to the rules. Nor can you document the test behavior in advance: It can only be known at runtime when the conditions are satisfied. Your test steps, while fewer, will be more complex, fragmented and difficult to understand and therefore tricky to maintain.

Reusing test steps as a precious commodity may make sense if it takes a costly programmer to develop them, but with a graphical data model like Certify it doesn’t. Copying, building or modifying steps– and especially maintaining them - is fast and easy enough that you don’t have to be afraid of a few extra. You trade redundancy for readability and predictability

And the fact is that the very definition of a test case is definitive: that is, the test must specify what is expected. Adding a bunch of “if, then and else” clauses introduces inherent unpredictability and introduces the possibility of the very logic errors that the application developers are subject to. When you don’t know what to expect, I maintain, it is no longer a test - it is an experiment.

Now, does that mean you will never have any logic? No.

The logic you still need is the intelligence to enable unattended test execution, so that if something happens that is unpredictable or unexpected, the test can compensate and perhaps continue. There are three basic types of execution risk:

Known risks are those you know exist and can remove through logic or other means. For example, if you can’t control or predict the test environment data - a different problem altogether - you may want to verify inventory to assure quantities are sufficient before you create a sales order. You know what the risk is and you remove it. Risks of this type, however, should really be solved at the root. For example, establish control and predictability of your test data so there would be no question.

Known unknown risks are those you know exist but can’t remove. Timing is a good example of this. You cannot predict nor control user response time or system execution time and so need logic to adjust to actual behavior. For example, waiting for the system to acknowledge that a transaction has posted so the test can proceed will likely require some logic to detect and wait for the response.


Unknown risks are the surprises – risks you don’t know exist but are possible. Errors can spring from beneath a number of rocks including the entire technology stack, data sources and of course the application under test. There is no way to predict every possible error from every player in the operating environment. For these risks, you need global recovery logic that returns the context of the test to a known place from which the next test may proceed. Without this basic logic, your first unexpected response means the end of the entire test session.

Now, you might hear “logic” and think programming. Not necessarily. In Certify, for example, complex decisions can be made by simply selecting from available options in a list. But just because it’s easy doesn’t mean it’s always a good idea. The less logic you write the better. So use it sparingly, only when you can’t remove the risk through other means. But when you do use it, try to make it as reusable and straightforward as you can. For example, create shared global recovery steps.

There are many other benefits to designing tests without logic. They can be used for documentation, training and compliance. Their readability makes them easier to understand, debug and maintain. Yes, maintain. Even though you may have more tests and steps overall, in a data model like Certify it is easy to locate steps that need changes and in most cases do global maintenance.

The key is to remember that test design should always focus on the expected and the unexpected should be minimized with proper test environment planning, setup and control. In the end, the best logic against using logic is that predictability trumps reusability.