Load Testing Blog
Functional test suites do NOT make good load tests!
Functional test suites do NOT make good load tests! Functional test suites have an inherently different purpose than performance test suites. It is always tempting to take your suite of functional tests and use that as the basis for your load test. This is especially true if your testing tool supports both functional and load testing.
The purpose of functional testing is to determine if the site works correctly. As every good tester knows, this means testing the edge cases. What happens when the user does something unexpected? What if I complete the “How many children do you have?” field by … Continue reading »
Choosing the User Scenarios to Load Test
The first step in building a load test is to choose the scenarios to test – this is frequently harder than one might guess. When we ask customers which scenarios we should test first, a common answer is “test them all”. While this might yield the most accurate results, it is rarely practical for the allotted budget and schedule. There is great value in getting load test results sooner rather than later – so we recommend testing as early as possible and taking an iterative approach.
Start with the most common operations that users will … Continue reading »
Assume nothing. Test everything!
Just because you are paying for a 100MB connection does not mean that you will get that much. And if you don’t test it, you will not know until it is too late.
Just because your admins insist that your test environment is an “exact copy” of your production site does not mean that they are. And if you don’t test it, you will not know until it is too late.
Just because your web server was the bottleneck in the last test and you added another front-end server and put a load balancer in … Continue reading »
Baseline the Performance of the Old System
If your project is replacing an existing system, it is immensely valuable to establish a baseline for the new system based on the existing system. Start by analyzing the usage patterns of the existing site. What operations are most common? What paths are users following through the site? How many users are accessing the system at various times throughout the day? Wherever possible, this data should come from system logs rather than assumptions and guesswork. Then start designing your test:
Create a mix of scenarios that account for 70-80% of the site usage
Add any other scenarios that are known, or suspected, … Continue reading »
3 Podium Finishes for the Web Performance Race Car
Web Performance is one of the sponsors, along with OpenNMS, of Jason Tower in his Spec E30 race car for 2010 and 2011. He’s consistently made progress as a driver, so that by the end of the season he was able to place third in all three races in October’s Great Pumpkin Run at the Carolina Motorsports Park!
A video of the first race is below, along with Jason’s blow-by-blow description:
“Saturday qualifying was smooth except for a GTS4 car that spun, I nabbed 5th of 17 cars with only a tenth of a second separating P3 and P5. … Continue reading »
The Fastest Webserver?
Looking for the snappiest, fastest web server software available on this here internet? So were we. Valid, independent, non-synthetic benchmarks can be difficult to find. Of course, we all know that benchmarks don’t tell us everything we need to know about real-world performance; but what’s the fun of having choices if we can’t pick the best?
Exactly. I decided to do a little research project of my own.
Test Plan
I selected for this exercise recent (as of October 2011) versions of Apache, Nginx, Lighttpd, G-WAN, and IIS — a list that includes the most popular web servers as well as web servers … Continue reading »
Systems never pass the first test
Out of our entire list of services customers, only a handful have satisfied their performance goals on the first test. Of those, all but one had been through a performance testing campaign with us in the previous year. If this is the first time for your project or your organization to undertake performance testing, you are virtually guaranteed to fail the first test.
There is a large list of things that can go wrong with modern web systems – firewalls, load balancers, databases, web servers and, of course, the code. One setting buried deep in an … Continue reading »
Tips for Implementing a Content Delivery Network
This post will focus on providing tips for implementing a CDN. Since there are multiple CDN providers, each with their own setup procedures, configuration of a CDN will be contingent on the provider and the service rendered. However, whether it be a CDN for a blog or website, the following pointers will be useful while implementing a CDN.
Ensure that YSlow recognizes the CDN. YSlow is pre-loaded with a list of popular Content Delivery Networks, however all CDNs are not included in the list. If a CDN is not included in the list, a site will most likely get an F … Continue reading »
Load Test preparation will take longer than you think
Our first-time services customers greatly under-estimate the time required to get the first test configured and ready to run. In these cases, they have employed us to design the tests and develop the testcases – so that part goes pretty quick. What they, and other first-time load testers, don’t account for is the amount of time required for their people to:
Decide what to test
Create a set of test data in the system (e.g. test accounts with usernames and passwords)
Install monitoring tools on servers
Implement/execute backup and restore procedures
Deciding what to test may require interviews with end-users, … Continue reading »
Load Test Often!
Having performance testing results from frequent points in the development timeline can help developers understand the performance impact of various code and system changes. When testing early in the development process on a test rig that is not equivalent to production, the performance numbers are not valuable in their own right, but _changes_ in the performance numbers can extremely valuable. These changes can reveal newly-introduced performance bottlenecks that should be investigated.
For each new development iteration, the previous test (which does not exercise the new functions) should be repeated to determine if the changes had an effect on the performance of … Continue reading »