Functional test suites do NOT make good load tests!
Functional test suites do NOT make good load tests! Functional test suites have an inherently different purpose than performance test suites. It is always tempting to take your suite of functional tests and use that as the basis for your load test. This is especially true if your testing tool supports both functional and load testing.
The purpose of functional testing is to determine if the site works correctly. As every good tester knows, this means testing the edge cases. What happens when the user does something unexpected? What if I complete the “How many children do you have?” field by entering 4,294,967,297? The more fields there are on a form, the more variations of data there is for the form – and thus there will be more tests for that form. As a result, a good functional test suite will not exercise the operations in a ratio that is consistent with real world usage, but rather in a ratio of the number of edge cases in a scenario.
That is very different from a good performance test suite. A good performance test suite attempts to model the real-world traffic patterns that the site is expected to endure. It contains the scenarios which account for 95% of the traffic on the website and may also include high-risk operations – back-ups, mass-edits, etc.
Now I’ll add my own edge case: If your functional testing tool is compatible with your load testing tool and you can simply push a button to run a test, load then this might not be a bad way to run your first smoke test. A smoke test is simply a load test that subjects the system to some amount of load just to see if it explodes in a cloud of smoke. The results should be interpreted as a binary – the system passed or failed. Because the test does not reflect real-world traffic patterns, these results should NOT be used to estimate capacity or judge the response times of the pages tested. However, it can be useful for finding out if the system is stable and explore the failure modes. It could reveal component limitations, load-balancer problems or other system-wide problems that should be caught and fixed as early in the testing process as possible. This is useful if (and only if) it allows you to run your first load test earlier in the development cycle (see my test early and test often tips). The downside of this approach is (1) it might send you off on a wild-goose chase, fixing problems that are unlikely to ever occur in real-world usage and (2) you are spending time you could have spent running more realistic load tests.
Chris Merrill, Chief Engineer