Recording in Load Tester: What’s Going On?
Load Tester is designed to recognize common use patterns and transform them into working test cases with a minimum of user intervention. Our clients appreciate that Load Tester does a tremendous amount of work automatically. But, any load testing tool must be prepared to exercise the complexity of its target, and for some web applications even a simple test case can involve hundreds of variables.
This post will try to elucidate the stages of the recording process, which are, roughly:
- Recording
- Initial Inspection
- Automatic Configuration
- Manual Configuration
- Replay
- Validation
During the recording stage, we manually perform a use case in a web browser. Load Tester monitors our work and, for the time being, only notes the literal transactions between the browser and the remote server. This leaves us with nothing more or less than a raw log of the interaction.
During the initial inspection we check that Load Tester has actually recorded what we want it to record. Why wouldn’t it? Load Tester may pick up irrelevant traffic, for example, if the browser’s RSS reader fires off during the recording (note that this occurrence will probably also disrupt Load Tester’s think time measurements). Alternately, the site may be deep-linking resources from another domain, which might concern us if we didn’t intend it. For example, I recently encountered a site that fetched a small XML file directly from the client’s subversion repository. In either case, we want to delete these references from our test case unless we have permission to expand our load test beyond our own network.
Examples of irrelevant traffic that might get picked up by Load Tester.
At this stage, we still have only raw transaction data. Load tester can now perform considerable automatic configuration for us. This is the job of the Application State Wizard, which parses the transaction log and generates a dynamic test case. Much of Load Tester’s analysis is so reliable that it happens entirely under the hood, but Load Tester also generates hundreds of variables that you can configure manually. These variables are lifted out of the prior transaction using extractors and inserted into the subsequent transaction using fields. Each extractor and each field explicitly names the variable it uses. It may help to know that all automatically generated variables begin with a ‘#’ sign.
Fields View (Before running the Application State Wizard)
Fields View (After running the Application State Wizard)
Extractors View (After running the Application State Wizard)
We rarely want to do significant manual configuration — Load Tester is very good at teasing out the flow of web applications. It will even detect most authentication schemes and configure them for us, requiring only an accurate table of username/password pairs. If necessary, we can define our own extractors using delimiters or regular expressions. Although it does not make sense to create new fields, we can redefine a field to any constant value, variable, or data table.
A standardized, populated web application should be predictable enough for Load Tester. For example, if our test cases pass with an unpopulated database but fail after we add the first element, this is because our web application behaves differently on that edge case — but there’s normally no reason to load test an unpopulated database. Even a site I tested with a very dynamic user interface only needed one regular expression extractor out of six test cases.
Occasionally we may find that the Application State Wizard is unnecessarily ambitious. We can rerun the wizard at any time, and ask it to ignore information that shouldn’t be managed dynamically.
How to instruct the Application State Wizard to ignore irrelevant transaction information.
Once we’re satisfied that our test case is correct, we can test it by replaying it. Inspect the results by viewing them in Load Tester’s built in browser. Replay a test multiple times. If you encounter an error, be aware that most errors have simple causes. Make sure your server is actually returning the page you expect it to return.
Finally, it’s a good idea to write validation rules for every page. For example, after I log in, I might expect to find the string “Welcome to This Web Site” somewhere in the content of the page. Load Tester automatically detects many errors, but we’ve found that liberal use of validation rules improves the quality of our test cases. Validators may apply to individual pages, transactions, or even to the entire test suite.