Load Testing with 5,000,000 Concurrent Users
Ever run a test with more than a million users? We’ve been getting requests for larger and larger tests recently with the upcoming elections, not to mention the trend for more online testing. Although we’ve been running large scale tests for over a decade, to make sure the test results are valid we are constantly doing internal testing. This ground work has already come in handy preparing for the 2016 election primaries, and we’re currently working to make sure state-level election websites are ready to go in November.
Web Performance Consulting
Doesn’t a CDN Solve all of My Problems?
Many of our customers are now using CDNs, reasoning that it will greatly increase their site’s capacity without a lot of trouble, and while that’s true, it also doesn’t happen automatically. Time and time again we’ve tested a newly CDN-enabled site only to discover that the site as configured was showing only a marginal performance increase, and certainly not enough to justify the price of a CDN. Luckily, testing showed what was slowing down the sites, and enabled configuration changes to make sure the CDN was delivering all of the performance gains it was capable of.
Configuring a Website to handle 5,000,000 Users
The first problem is testing millions of concurrent users is fairly expensive, but we’ve invested the time and money to insure large tests go smoothly and give accurate results, creating thousands of computers in the cloud (over and over again) to generate up to 5,000,000 concurrent users at last count. The second problem is in order to test such a large load you of course need to host sites big enough to withstand the onslaught, but its necessary to make sure we’re ready to go when you need large amounts of traffic. NGINX has been invaluable in this regard, which is incredibly fast for the static websites we use for this type of large-scale testing.
The first level of testing we do for very large scale tests is can the requested load be generated accurately? The following chart shows that the 1,200 load generators were able to smoothly create and sustain users at every level up to 5,000,000. Note that the load generators are monitored for CPU and memory usage, and thus won’t add users unless there are enough system resources.
What does 5,000,000 Concurrent Users Really Mean?
One of the big questions in capacity measurement is how to translate between hard measurements like bandwidth and hits/sec into concurrent users. After all, 100 people clicking on a new page every 10 seconds creates twice as much traffic as 100 people clicking on a new page every 20 seconds. For new sites this is just a guess, but for existing sites, a careful analysis of the analytics greatly increases test accuracy. For example, Google Analytics keeps track on how long people spend on each page, and for webperformance.com the average time is around two minutes:
The overall “think time” for each page in a test case would then be set for two minutes, while the software varies this by +/- 25-50%. For important pages, such as forms and the home page, you can custom-set the think times if they vary much by the site average. In the test I just ran this week to try out 5,000,000 users, I wanted to try and push the think times to the lowest practical value that might be seen in the field, which translated into a maximum of 1,300,000 pages per second or roughly 4 seconds spent on each page, which is really fast compared to live commercial sites.
The website being tested, in this case based on NGINX, is pretty darn fast with static content, which gave incredibly fast sub-second response times for testing purposes.
The other factor to look for is bandwidth, and each load generator had 100Gbps available, giving us a smooth bandwidth usage profile out of the web servers.
Currently any number of concurrent users higher than 500,000 is handled by our services department, and thanks to our experience is very competitively priced. Let us know if you need any help preparing your massively scalable web-based application.