Testing tip: Make it Fail
We recently had a case of debugging a testcase where Load Tester reported an error with a page that looked and appeared just as it should, except it was missing a link the user needed to move forward. Debugging this case was easy: examine the replay two pages prior to the error, and the server had added a red error message to the page indicating the selected criteria was not valid.
So why didn’t Load Tester flag this error from the server? Beneath the error message, the page was exactly identical to the expect page: form fields were present and filled out. The HTTP Status Code was also OK. The error message itself was inserted by the server as decorative markup, and Load Tester will generally ignore minor markup changes.
This case is not too bad since the testcase failed, and gave the user a problem to diagnose and correct. But what would happen if the error occurs at the end of a test, when the testcase is being finalized or committed? Unfortunately, this is often the case for many systems. In purchasing systems, it is not uncommon to complete the sale, and get a receipt back from the system (containing all the expected items), but with an extra blurb of red text at the top: “An error occurred, please contact…”. If this happens in a Load Test, and Load Tester was not configured to catch the error, then Load Tester would not indicate that the site was experiencing functional trouble.
The best way to avoid this behavior is during the test development procedure: force the tests to fail. This is most important towards the end of the test – an invalid login may not cause errors immediately, but most systems will eventually make it clear that something is wrong when a user is not logged in and able to gain access to the necessary pages. An easy way to test failure detection is to create a dataset with invalid data. Use the Fields View to locate and change fields where user entered data or selected action identifiers are being passed back to the application.
Now, when replaying the testcase, check to see that the server is generating an error message. The objective is not only for Load Tester to indicate that an error has occurred, but also to ensure that Load Tester reports an error on the same page where the server generated an error message. If the server does not generate an error message for invalid user data, then you have another problem.
You may be wondering: why look for an error message indicating invalid data, when I know all the data will be valid during the test and there is no reason the server should ever generate the same error? Answer: while the error message may be unique to the error you’ve introduced, the markup around error messages is often generic. The goal is to examine the error message, and look for markup or common text that the application is likely to reuse for any error message. Once you know the general cases, you will want to create Validation Rules for your system. While error markup often varies between systems, here are some common examples:
MIME contains: text
Flags any paragraph or new block of text starting with the word “Error”.
MIME contains: text
MIME contains: text
This is useful for applications developed in .NET. This string will appear in stack traces, which some applications will display to the user, but others may hide in a HTML comment.
MIME contains: text
For .NET applications using AJAX, this delimited message is common to indicate that an AJAX request has failed.
MIME contains: text
This is useful for applications developed in Java. This string will appear in stack traces, which some applications will display to the user, but others may hide in a HTML comment.
MIME contains: html
Some applications will wrap error messages in a <div> or <span> tag, with a CSS class name like “error” or “errormsg”. This case will detect if any such tags exist. It does not, however, detect if there is any text within the tag – which may be necessary for applications which always generate an error block but may leave the text empty when there is no error to report.
MIME contains: html
For Sharepoint applications, this is a common indicator that a Sharepoint error message will appear in the page
Once you’ve written your Validation Rules, you will need to re-run the Validators Wizard (right click on your testcase and select Configure » Validators) to apply the rules to your testcase. The rules should be fine tuned until Load Tester correctly identifies not that a specific data error was present, but the more general problem: the page contained an error message. Once you are satisfied that Load Tester is able to identify your application’s custom error messages, don’t forget to change any testcases back to their functioning version and ensure no further errors are being generated.
I hope this helps, but if you have any questions, our friendly support staff is willing to assist in getting your tests working for you.
Happy Testing!
-Frank
Engineer at Web Performance