Why Is Most Testing Still Manual?
Despite decades of investment in test automation tools, techniques and time, the majority of enterprise application testing is still performed manually. I have always believed it is because the test tools were too technical and time-consuming to use, and if we could only make them easy and accessible then things would change. And while there may be some truth to this, it is not the whole story.
To understand the whole story, you have to step back and look at why testing happens, who does it and what this means to automation.
Why Test?
Testing happens because something is changing. In most enterprises, there are two streams of changes. One is the stream of projects, usually planned and executed within functional silos to advance specific goals. The other is maintenance or sustaining, which responds to production issues or sudden business demands. In this latter stream, changes may happen daily or even more often. These typically unplanned, out of window changes are a subject unto themselves for the next blog installment. For now let’s focus on planned projects.
When a project is planned, an allocation of time and resources is made to testing based on scope and risk. Of course even planned projects can result in both expected and unexpected outcomes, but the scope constraint imposed by the project necessarily focuses attention on verifying the expected results. Potential unexpected results are so undefined and broad in scope that they are either ignored or given cursory attention in the form of minimal regression testing. This means that most project tests are tightly focused around the specific changes being introduced.
This focus on what is changing makes sense from a resource and time allocation point of view, but for automation it’s bad news. The reason is that highly focused tests have limited reusability as a practical matter.
Here’s an example: Let’s say the enterprise has established a new distribution center that services a particular line of products and geographic area, and a project is launched to implement it and all its attendant business rules within the enterprise systems. As you would expect, the project calls for tests that verify that indeed the new center appears in the system and that its products can be shipped to the areas it services.
And while this all makes sense within the context of the project, it makes less sense to automate these tests for the simple reason that once the distribution center is operational, future projects will likely focus on something else and there will be no reason – and no resources – to keep running the same tests. Even if the tests are automated, why would they be re-executed and who would do it? Why would that particular distribution center be any more important than any other?
In this case, it’s not the ability or technology to automate that is missing – it’s the lack of interest in reusing the tests. And that lack of interest is only worsened by the way projects are organized.
Who Tests?
Projects are, by their nature, transient; they have a limited scope and duration. They are formed, executed, then concluded. The test resources assigned to enterprise application projects are typically business process experts borrowed for the duration from the operational area and functional silo that initiated the project. Thus, project test team members are also transients and scatter to other projects or back to their day jobs when it is concluded.
The upshot is that new project members may be unfamiliar with the artifacts and tools left behind by any previous project, or uninterested in them because of their irrelevance to the planned changes.
Unfortunately, this discontinuous approach to changes results in a low level of reuse, even when it might be justified. For example, I am familiar with a project that flatly refused to even explore, let alone use, an extensive library of automated tests created by a previous project team on the grounds that the previous project was concerned with a completely different set of changes and therefore the tests were irrelevant to them and not worth the effort of engaging the extra skill sets needed to take advantage of them.
Even documented manual tests may fall into disuse. Ironically, the more tests that are accumulated the less likely they are to be reused. New project members may not have the time or motivation to review hundreds or even thousands of previous tests to look for relevance; it is often easier and faster to just create something new than to locate and understand something old.
Who Cares?
In the end, manual testing predominates in this environment because there is little to no incentive – or budget or schedule – within most projects to invest in reusability. The project team is constrained by its own expertise and logistics to focus on the functionality at hand, with limited if any understanding of the potential impact on other areas or future projects.
Clearly there are some projects that are broad enough in scope to warrant automation, such as an initial roll-out or an upgrade that has wide-ranging potential impact. And just as clearly there are some aspects of almost every project that would justify reuse on the grounds that they are adding to the overall functional base and therefore represent a new component of risk. But the problem is that the transient nature and structure of the project itself does not lend itself to investing in reusability.
What is especially ironic about this conclusion is that in most enterprises, the only formal testing that ever happens is inside the project stream, yet the majority of changes actually come from the sustaining stream where testing is minimal and informal at best. But that’s a whole other subject for the next installment.
To understand the whole story, you have to step back and look at why testing happens, who does it and what this means to automation.
Why Test?
Testing happens because something is changing. In most enterprises, there are two streams of changes. One is the stream of projects, usually planned and executed within functional silos to advance specific goals. The other is maintenance or sustaining, which responds to production issues or sudden business demands. In this latter stream, changes may happen daily or even more often. These typically unplanned, out of window changes are a subject unto themselves for the next blog installment. For now let’s focus on planned projects.
When a project is planned, an allocation of time and resources is made to testing based on scope and risk. Of course even planned projects can result in both expected and unexpected outcomes, but the scope constraint imposed by the project necessarily focuses attention on verifying the expected results. Potential unexpected results are so undefined and broad in scope that they are either ignored or given cursory attention in the form of minimal regression testing. This means that most project tests are tightly focused around the specific changes being introduced.
This focus on what is changing makes sense from a resource and time allocation point of view, but for automation it’s bad news. The reason is that highly focused tests have limited reusability as a practical matter.
Here’s an example: Let’s say the enterprise has established a new distribution center that services a particular line of products and geographic area, and a project is launched to implement it and all its attendant business rules within the enterprise systems. As you would expect, the project calls for tests that verify that indeed the new center appears in the system and that its products can be shipped to the areas it services.
And while this all makes sense within the context of the project, it makes less sense to automate these tests for the simple reason that once the distribution center is operational, future projects will likely focus on something else and there will be no reason – and no resources – to keep running the same tests. Even if the tests are automated, why would they be re-executed and who would do it? Why would that particular distribution center be any more important than any other?
In this case, it’s not the ability or technology to automate that is missing – it’s the lack of interest in reusing the tests. And that lack of interest is only worsened by the way projects are organized.
Who Tests?
Projects are, by their nature, transient; they have a limited scope and duration. They are formed, executed, then concluded. The test resources assigned to enterprise application projects are typically business process experts borrowed for the duration from the operational area and functional silo that initiated the project. Thus, project test team members are also transients and scatter to other projects or back to their day jobs when it is concluded.
The upshot is that new project members may be unfamiliar with the artifacts and tools left behind by any previous project, or uninterested in them because of their irrelevance to the planned changes.
Unfortunately, this discontinuous approach to changes results in a low level of reuse, even when it might be justified. For example, I am familiar with a project that flatly refused to even explore, let alone use, an extensive library of automated tests created by a previous project team on the grounds that the previous project was concerned with a completely different set of changes and therefore the tests were irrelevant to them and not worth the effort of engaging the extra skill sets needed to take advantage of them.
Even documented manual tests may fall into disuse. Ironically, the more tests that are accumulated the less likely they are to be reused. New project members may not have the time or motivation to review hundreds or even thousands of previous tests to look for relevance; it is often easier and faster to just create something new than to locate and understand something old.
Who Cares?
In the end, manual testing predominates in this environment because there is little to no incentive – or budget or schedule – within most projects to invest in reusability. The project team is constrained by its own expertise and logistics to focus on the functionality at hand, with limited if any understanding of the potential impact on other areas or future projects.
Clearly there are some projects that are broad enough in scope to warrant automation, such as an initial roll-out or an upgrade that has wide-ranging potential impact. And just as clearly there are some aspects of almost every project that would justify reuse on the grounds that they are adding to the overall functional base and therefore represent a new component of risk. But the problem is that the transient nature and structure of the project itself does not lend itself to investing in reusability.
What is especially ironic about this conclusion is that in most enterprises, the only formal testing that ever happens is inside the project stream, yet the majority of changes actually come from the sustaining stream where testing is minimal and informal at best. But that’s a whole other subject for the next installment.
posted by WorkSoft
at
1:58 PM
1 Comments
