Identify Performance Goals Early…and Measure Them!
When a load test is complete, you will be asked “How did we do?” Do you know how will you answer that question? Our customers come to us and know, for example, that they need their site to handle 1000 users, but they frequently cannot tell us what “handle 1000 users” means to them. You will need to know which metrics are important and what the goals for those metrics are – preferably long before you start testing.
The first step is to determine what you should be measuring. For websites, you will typically be interested in page duration – how long it takes to display a page in the browser. This will usually be associated with another metric for the amount of load – the most common is simultaneous users. Other common metrics for load are hits/sec, transactions/min and page-views/hour.
Once you know what should be measured, you must assign goals to them. For example “All pages must load within 4 seconds under a load of 1000 users.” Without specific, concrete goals, you will not be able to give a specific, concrete answer to the “How did we do?” question.
Next, look for exceptions to these goals. Are there some pages that are ok to be slower, such as the final check-out page in a shopping cart or a site-wide search? Are there some that should be faster, such as the “Add to cart” button?
Knowing what to measure and what the goals are will allow you to design tests to accurately assess the system’s ability to meet those goals.
Chris Merrill, Chief Engineer