Load Tester improvement – better success and fail metrics
Previous versions of Load Tester had 2 metrics describing the success or failure of a web page (or single transaction) during a load test: Repeats and Errors. A Repeat was counted any time a valid HTTP response was received from the server. Errors, however, were a bit more complex. Errors could occur at any point in the transaction, from establishing a connection to post-response validation. An error was counted if the connection was terminated in the middle of a transaction. It was also counted if the status code from the server indicated a failure (e.g. a 500 status). Validators record an error when the expected content was not received from the server. Extractors, which search for dynamic content on a page that is needed later in the testcase, record an error when the required content is not found. This could result in several errors reported for a single web page — especially since every transaction within a page can generate one or more errors. This made it impossible to determine exactly how many pages were attempted or failed during a given period in a load test.
The 3.6 release of Load Tester will include 4 new metrics, which replace the Repeats and Errors metrics. The new metrics are Successes, Failures, Completions and Failure Rate.
- Successes – A success is counted each time a page completes with no errors
- Failures – A failure is counted each time a page completes and at least one error was encountered
- Completions – A completion is counted each time a valid HTTP response is received from the server
- Failure Rate – the ratio of Failures to total attempts (Successes + Failures)
With these new metrics, it is easy to see exactly how many pages succeeded and failed, as well as the total number of requests served (Completions) and the total number of attempts (the sum of Successes and Failures). The Failure Rate is clearly shown for each page and transaction – and is shown by default in the displays and reports wherever the Errors metric was shown in previous versions.
Note that each error encountered (network, validation, etc) is still recorded as an individual entity, but they are not counted – since the number of errors is not generally valuable (for the reasons mentioned above).
We hope this will clear up confusion faced by some users in previous versions of Load Tester!