The successful adoption of automated tests
Many companies decide to use automated tests in their software quality assurance. This is nothing new, but it became more and more important as more tools appeared on the market to assist these attempts. They are all “easy to use” and the “solution to all problems of testing” and can “prevent bad software quality”. But let’s have a closer look at this.
If “a company” decides to use a test automation tool, it’s in general a management decision as this is an investment. You need people to evaluate the right tool (or at least, you should do this!), a tool must be purchased (if it is not open source), test cases need to be automated and you’ll need an environment to execute these automated tests. A very common cause for the decision is the idea to reduce time and expenses for testing. This would imply that you can save money for your testing staff and gain a better time-to-market. At first glance this looks like a good idea. Automated tests are quicker than manual tests, so this can be achieved. But why not just run fewer tests, avoid long test runs and reduce regression testing? This would be the easiest way to save time and money. Well, the most certain answer would be: “Because our software would be full of bugs and our customers will be very dissatisfied with it”. So it is quite obvious, that testing is not the problem, but the quality of the software. So at first you should have a look at your software developing process and your test process and optimize them.
So, if test automation is not the solution to all problems in testing, what is it good for? Well, first of all it’s good for regression testing. Every tester knows how boring it is to test the same stuff every release again and again. And humans tend to work sloppy, when things get boring. But this is a perfect task for a computer. It does not bother to do the same things again and it runs tests in the 500th iteration as accurate as in the first and is still faster than a human tester. With a good automated regression test you can release your tester from this unloved task and I’m sure there are a lot of other things to test for them.
Did I say “a good automated regression test”? What is “good” in this case? How do I know that my test automation helps me? Or if I try to improve it, how can I know, that it has improved? The answer to this is easy: Measuring. And for that we need metrics. One easy to calculate metric for your test process is the DDP (defect detection percentage). It’s just the defects found by testing divided by the total number of known defects. This is a good indicator to monitor improvements in your testing process, but it should not be the only one.
With the help of metrics, it is possible to monitor your testing process, but the test automation is only a part of this. So, which quality attributes are important for test automation? Although this depends on your goals, your environment and your software etc. there are some general attributes, which should be kept in mind:
What are we doing here at ICANS? We automated every business critical part of PokerStrategy.com. Those tests are running every night on different test stages. We have about 865 test cases in our GUI-based test automation, these tests run about 9 hours. The test execution is performed with 5 virtual Windows Machines with Firefox and Internet Explorer. This way we were able to cut our manual regression test from 12 hour per release to 4 hours per release although we’re testing a lot more than before. All test results can be viewed in our continuous integration system “Jenkins”. By this we have a good transparency of the current status of our software quality. We can react fast on defects and have the good feeling to know, that we did not forget to test something important.