Load Testing Blog
Validate the Testcases
When a website is placed under heavy load, it can fail in a wide variety of ways. The site may become unresponsive or return an HTTP status code indicating an error (500). Or it might return a web page that is mostly complete (the headers, footers and navigation are all present) but is missing a critical part that was related to the failure.
If the load test scenarios (testcases) do not validate page content well enough to catch these problems, the results could be misleading. I.e. the system could be failing but the results … Continue reading »
Test Only as Accurately as Needed
In previous posts, I’ve talked a lot about the dangers of not accurately simulating the real-world usage. But there is another side to this story: obsessing too much over an accurate simulation can be very costly. There is one very important thing to understand about load testing: No matter how hard we strive to make tests accurate, they are still only an estimate.
Your load test will never match even one single hour of production usage, not even once, no matter how hard you try. Bold statement? Perhaps. Consider this: even … Continue reading »
Identify and Test High-Risk Operations
As I described in my previous post on choosing test scenarios, the early tests should focus on the high-traffic scenarios. After testing and optimizing those, the next step is to identify and test the rarely-used but high-risk operations. These are operations that may have an impact on performance that is not proportional to their frequency. Even though these are not as common, they can have large impacts due to their system-wide nature and can cause sporadic performance drops during production hours that defy explanation.
Examples include:
Complex searches
Updates that touch many joined … Continue reading »
Design a Load Test that Measures What You Really Want to Know
When the first few testcases are ready, the next step is to put them together into a load test. How they are combined and executed will have a big impact on the accuracy of the test.
Testcase ratios – Make sure that each testcase is exercised in the correct ratio relative to the other testcases. For example, it is likely that many more people will be searching for products than buying them – the load configuration should reflect this ratio.
Page ratios – Re-visit the hit … Continue reading »
Build Realistic Testcases
When you have identified the first group of scenarios to test, it is time to turn them into testcases, which is the term we use for the representation of the scenario in software. In it’s simplest form, a testcase is a list of pages to visit. In the simplest tools, it may simply be a list of URLs. With most tools, the testcase is a program generated by the tool and customized by the tester. More sophisticated tools, such as Load Tester, present the testcase … Continue reading »
Functional test suites do NOT make good load tests!
Functional test suites do NOT make good load tests! Functional test suites have an inherently different purpose than performance test suites. It is always tempting to take your suite of functional tests and use that as the basis for your load test. This is especially true if your testing tool supports both functional and load testing.
The purpose of functional testing is to determine if the site works correctly. As every good tester knows, this means testing the edge cases. What happens when the user does something unexpected? … Continue reading »
Choosing the User Scenarios to Load Test
The first step in building a load test is to choose the scenarios to test – this is frequently harder than one might guess. When we ask customers which scenarios we should test first, a common answer is “test them all”. While this might yield the most accurate results, it is rarely practical for the allotted budget and schedule. There is great value in getting load test results sooner rather than later – so we recommend testing as early as possible and taking an … Continue reading »
Assume nothing. Test everything!
Just because you are paying for a 100MB connection does not mean that you will get that much. And if you don’t test it, you will not know until it is too late.
Just because your admins insist that your test environment is an “exact copy” of your production site does not mean that they are. And if you don’t test it, you will not know until it is too late.
Just because your web server was the bottleneck in the last test and you added another front-end … Continue reading »
Baseline the Performance of the Old System
If your project is replacing an existing system, it is immensely valuable to establish a baseline for the new system based on the existing system. Start by analyzing the usage patterns of the existing site. What operations are most common? What paths are users following through the site? How many users are accessing the system at various times throughout the day? Wherever possible, this data should come from system logs rather than assumptions and guesswork. Then start designing your test:
Create a mix of scenarios that account for 70-80% of … Continue reading »
Systems never pass the first test
Out of our entire list of services customers, only a handful have satisfied their performance goals on the first test. Of those, all but one had been through a performance testing campaign with us in the previous year. If this is the first time for your project or your organization to undertake performance testing, you are virtually guaranteed to fail the first test.
There is a large list of things that can go wrong with modern web systems – firewalls, load balancers, databases, web servers and, … Continue reading »
