Do the Math: Automation is No Longer Optional
I don’t get it. Software development has evolved from a slow, painstaking, manual discipline to a rapid, iterative, highly automated process. Computers have escaped glass-walled rooms with raised floors to land on palm-sized devices. Applications have shifted from back-office productivity aids to front line competitive weapons.
Yet the majority of functional testing is still done manually.
It’s not for lack of technology – automation tools have been around for decades. I should know, I co-founded a record/play/script company in the 1980’s called AutoTester and followed up with Worksoft to take the coding out of automation. And of course the test tool ecosystem has expanded exponentially with a broad range of companies and products.
Neither it is a lack of need. The graph below is my favorite because it says it all: over time, functionality grows but testing resources don’t. Unless you can employ automation to create a cumulative library of tests, you are doomed to steadily decrease coverage and increase risk with each succeeding release.
The risk is also exploding. Whereas before software failure likely meant that the accounting staff had to revert to calculators for a day or two, today errors can instantaneously affect thousands or even millions of customers, tainting the company’s brand, reducing revenue and driving up costs. In some infamous cases, inadequately tested software has cost hundreds of millions, even billions, of dollars.
So what gives?
As far as I can tell, it’s a combination of four factors:
1. An undefined test process. You can’t automate what isn’t defined, and manual tests are by their nature informal, usually relying on the knowledge and skill of the individual tester. If you don’t have a handle on what you need to test, you can’t predict the time and effort for automation. See “The Importance of Testing Software Requirements” http://searchsoftwarequality.techtarget.com/news/column/0,294698,sid92_gci1275907,00.html for a deeper dive into this topic.
2. Lack of software testability. Many modern application interfaces are dynamically generated on the fly, resulting in window and objects that have no useful or persistent names. A lack of testability standards may also cripple automation by presenting components that tools can’t interact with. These combine to make automation either impossible or so time-consuming that the payback isn’t there. See “Automated Testability Tips” http://www.stickyminds.com/s.asp?F=S6013_COL_2 for more.
3. Inadequate resources. Too many companies think that all they need to do is write a check to the tool vendor and automation is on the way. But buying a tool is like joining a health club: you still have to invest the effort. And you can’t expect existing staff to keep up with manual testing while automating in their spare time. See “Join the Club” http://www.stickyminds.com/s.asp?F=S10695_COL_2
4. Unrealistic expectations. Record and play demos paint a false picture of what automation is all about, so management believes that it requires nothing more than capturing the manual process and therefore expects immediate results. This leads to a failure to instill development disciplines and allocate the proper time, attention and resources to getting the process defined and automated. See “The Demise of Record Play Script” http://www.worksoft.com/pdf/TheDemiseofRecordScriptPlay.pdf for a better understanding of the true costs of record and play.
So what to do?
First of all, face up to the fact that manual testing is no longer a viable option. Don’t get me wrong – you will never automate 100% of your testing, simply because some tests are not worth automating and others, like usability tests, depend on actual user interaction. But the vast majority of functional testing can, should – even must – be automated in order to achieve anything close to comprehensive coverage on time and within budget.
Second, develop and present a reasonable plan for automation that takes into account the current state of your test process and sets realistic expectations. Accept a strategy for gradual progress, and reject any plan that fails to account for the additional time and effort that will be required to get established. Granted, automation saves time and resources, but that’s only after you get it into place.
Third, don’t expect to exclude your experts. Unless you have clear, current and comprehensive test case documentation (and who does?), you will need access to your best and brightest to help you identify the most critical aspects of the application and the business rules that govern them.
And fourth, attack the test data and environment issue head-on. Simply put, automation requires repeatable, stable data, and this is one of the most challenging components of any automation strategy. In fact, it’s such an important topic that it deserves its own separate treatment.
So stay tuned.
Yet the majority of functional testing is still done manually.
It’s not for lack of technology – automation tools have been around for decades. I should know, I co-founded a record/play/script company in the 1980’s called AutoTester and followed up with Worksoft to take the coding out of automation. And of course the test tool ecosystem has expanded exponentially with a broad range of companies and products.
Neither it is a lack of need. The graph below is my favorite because it says it all: over time, functionality grows but testing resources don’t. Unless you can employ automation to create a cumulative library of tests, you are doomed to steadily decrease coverage and increase risk with each succeeding release.
The risk is also exploding. Whereas before software failure likely meant that the accounting staff had to revert to calculators for a day or two, today errors can instantaneously affect thousands or even millions of customers, tainting the company’s brand, reducing revenue and driving up costs. In some infamous cases, inadequately tested software has cost hundreds of millions, even billions, of dollars.
So what gives?
As far as I can tell, it’s a combination of four factors:
1. An undefined test process. You can’t automate what isn’t defined, and manual tests are by their nature informal, usually relying on the knowledge and skill of the individual tester. If you don’t have a handle on what you need to test, you can’t predict the time and effort for automation. See “The Importance of Testing Software Requirements” http://searchsoftwarequality.techtarget.com/news/column/0,294698,sid92_gci1275907,00.html for a deeper dive into this topic.
2. Lack of software testability. Many modern application interfaces are dynamically generated on the fly, resulting in window and objects that have no useful or persistent names. A lack of testability standards may also cripple automation by presenting components that tools can’t interact with. These combine to make automation either impossible or so time-consuming that the payback isn’t there. See “Automated Testability Tips” http://www.stickyminds.com/s.asp?F=S6013_COL_2 for more.
3. Inadequate resources. Too many companies think that all they need to do is write a check to the tool vendor and automation is on the way. But buying a tool is like joining a health club: you still have to invest the effort. And you can’t expect existing staff to keep up with manual testing while automating in their spare time. See “Join the Club” http://www.stickyminds.com/s.asp?F=S10695_COL_2
4. Unrealistic expectations. Record and play demos paint a false picture of what automation is all about, so management believes that it requires nothing more than capturing the manual process and therefore expects immediate results. This leads to a failure to instill development disciplines and allocate the proper time, attention and resources to getting the process defined and automated. See “The Demise of Record Play Script” http://www.worksoft.com/pdf/TheDemiseofRecordScriptPlay.pdf for a better understanding of the true costs of record and play.
So what to do?
First of all, face up to the fact that manual testing is no longer a viable option. Don’t get me wrong – you will never automate 100% of your testing, simply because some tests are not worth automating and others, like usability tests, depend on actual user interaction. But the vast majority of functional testing can, should – even must – be automated in order to achieve anything close to comprehensive coverage on time and within budget.
Second, develop and present a reasonable plan for automation that takes into account the current state of your test process and sets realistic expectations. Accept a strategy for gradual progress, and reject any plan that fails to account for the additional time and effort that will be required to get established. Granted, automation saves time and resources, but that’s only after you get it into place.
Third, don’t expect to exclude your experts. Unless you have clear, current and comprehensive test case documentation (and who does?), you will need access to your best and brightest to help you identify the most critical aspects of the application and the business rules that govern them.
And fourth, attack the test data and environment issue head-on. Simply put, automation requires repeatable, stable data, and this is one of the most challenging components of any automation strategy. In fact, it’s such an important topic that it deserves its own separate treatment.
So stay tuned.
posted by WorkSoft
at
11:28 AM
2 Comments
