Validate the Testcases
When a website is placed under heavy load, it can fail in a wide variety of ways. The site may become unresponsive or return an HTTP status code indicating an error (500). Or it might return a web page that is mostly complete (the headers, footers and navigation are all present) but is missing a critical part that was related to the failure.
If the load test scenarios (testcases) do not validate page content well enough to catch these problems, the results could be misleading. I.e. the system could be failing but the results indicate success. In the process of helping our software customers understand their test results, we have come across this problem many times. The solution is simple – validate the pages.
Most load testing tools include some level of automatic validation. The most common is checking the HTTP status code. Unfortunately, many systems will still return a 200 status code (success) when an error has occurred. For most testcases, our product (Load Tester) automatically looks for a number of elements on a page and will indicate a failure if those are missing.
Regardless of what tool you are using, it is prudent to add some validation to ensure the pages coming back from the server are complete and correct. For example, if your site includes the username in the header, but not on error pages, then a simple check for the username on a page is a good start. If you know the page after the user finishes the checkout process includes the phrase “Your order has been processed”, then checking for that text will help catch order processing failures.
The conclusions derived from your load testing efforts can only be as good as the data – if the data is bad, the conclusions likely will be as well. So always validate your pages to ensure valid test results!
Chris Merrill, Chief Engineer