When to Automate Tests
Do not be fooled — test automation is not always superior to manual testing. It is not practical to automate all tests. Even if it were, some test cases are better suited towards manually testing. How do you know which tests to automate? Consider the following properties.
Re-Usability
Even if the test infrastructure is already available, developing a test script takes time. If a test only needs to be performed a few times it is more time-efficient to run it manually.
A perfect candidate for an automated test is any regression test. Regression tests confirm that an existing feature still operates correctly after a new code change. For teams applying continuous integration practices, regression tests should run whenever a commit is made to a repository. When several commits are made per day by each developer manual testing is no longer feasible.
Test Maintenance
Like any other form of code, test scripts and frameworks require maintenance. Even highly re-usable tests will need to be refactored as the project under test evolves. The added time to maintain a test case should be considered before it is created.
Run Time
Not all test cases are instantaneous. Some can take hours, or even days. Test cases with long run-times, or significant down time between steps should be automated.
How do you automate long test suites? If the test cases require hardware or other resources that could be used for development, schedule them to run outside of work hours. A simple scheduler that queues jobs and runs them later is all it takes.
Saving human time through tests scripts is valuable, but testing efficiency does not end at automation. Take the time to make your test suites run quicker. Faster test suites allow you to perform more tests to uncover more bugs before your code is released to production.
How do you speed up your test suites? Make them run in parallel via multithreading. Say you have 10 tests that take an hour each to run. On a single thread it will take 10 hours. If each test has its own thread it will only take 1 hour — that is 9 hours of testing time saved.
Amount of Data
If the test case uses large amounts of data or performs a very repetitive task — such as during stress testing — it should be automated. Stress testing is used to determine the breaking point of a system by putting it under extreme conditions. If you are testing a web application you can perform a stress test that floods it with POST or GET requests until it crashes. Performing this test even once manually would be too tedious and time consuming.
Pass/Fail Objectivity and Determinism
Automated test cases require objective pass/fail conditions, such as comparing values against a threshold or perform equality checks. If your test is subjective in any way it cannot be automated. A textbook example of a test case that should never be automated is anything related to the user experience. No matter how sophisticated your test is it can never substitute the opinions of your users.
Even if your test has objective pass/fail conditions you must also ensure it is deterministic, meaning that running it with the same inputs will always produce the same output. If your test flickers — passes sometimes and fails other times without changing the code under test — the test result no longer ensures that no bugs have been introduced. Instead, you will have to manually inspect each test run, defeating the purpose of automation.
Exhaustivity
Test scripts will only perform the checks they are told with the parameters they are given. If anything fails in between the test will still pass. Many different parameterizations can be implemented to increase exhaustivity, but what the test checks for is set. The tester has to assume how the system might fail before the test case is created, allowing for unforeseen bugs to go unnoticed.
Another type of test that cannot be automated is ad hoc testing. During ad hoc testing, the tester performs random checking. There is no pre-set testing plan — everything is improvised. Automated tests are not exploratory, and will never be able to find corner cases like a manual ad-hoc tester can.
Conclusion
Automated testing is a double edged sword. It can improve your quality assurance practices or be a detriment. Before you automate a test case, take the time to determine if the benefits outweigh the costs.
