Generating 1,000,000 Concurrent Users with Load Tester 5 - Web Performance
Menu

Generating 1,000,000 Concurrent Users with Load Tester 5

Can Load Tester really generate 1,000,000 concurrent users?  Many of our existing customers have run tests in the 20,000-50,000 range. With previous versions of Load Tester and some internal tools, our services group has run tests as large 600,000 VUs. But none of these tests really pushed Load Tester to it’s limits. We’ve engineered Load Tester from the start to scale, and scale big. All indications made us confident that our load generation infrastructure could scale far beyond any test we had run. However, the UI was not quite up to the task. Controlling the large number of load engines required for such a test stressed the command and control framework. That kicked back into the UI, making the user experience unpleasant (at best).

But before proceeding, we needed a goal. Very few sites can handle one-hundred-thousand simultaneous users, so I suggested that was a good goal. I proposed that we make the necessary improvements to make the user experience as pleasant at 100,000 users as it is at 10,000. But the marketing guy would have none of that. Michael wanted a big, round number. I asserted that 100,000 was both round and big, but I was overruled. The goal was set at 1,000,000 users. BTW, Michael is an engineer at heart and by degree, but we don’t let him do that much these days 😉

Next, we needed a test rig. Building a site that can serve 1,000,000 simultaneous users is not a trivial task. But Matt, our Senior Test Engineer and part-time server guru, came through with an EC2 image that could serve static content to 10,000 users at a pretty good clip, using NginX. Firing up 100 servers in Amazon’s cloud would give us a site to beat on.

Then it was time to run a few tests, just to see where we stood. We used only the tools that Load Tester users would have at their disposal  – Load Tester 4.3 and our cloud engines in EC2. Controlling 200 load engines (400,000 virtual users) was not too difficult. But after that point, the user experience started to degrade. The time required to upload the testcases to such a large number of engines was annoyingly long. And by the time we got up to the goal of 1,000,000 virtual users, the UI was pretty much unusable. However, the test completed successfully and the UI recovered (eventually). Lane volunteered to dive into the guts of the command and control code to take a look. He emerged a few weeks later with a long list of small tweaks and optimizations that added up to a huge improvement for the user experience.

The result is Load Tester 5 – which we have tested up to 1,000,000 users (well, actually 999,977 users). At that level, Load Tester was churning out load that would make most web admins weak in the knees:

Now, there is a caveat: our testcases were a little on the light side because we did not want to overload the web servers. Such an overload generally makes things easier on the load engines, since they would be sending and receiving less traffic. So the simulated users were visiting 10 relatively small pages (~55kB avg) over the course of several minutes. But – our load engines were also running well below capacity. The load engines have a capacity-estimation algorithm they run constantly and it is pretty conservative. The engines were reporting they could run 50% more users. If that estimate turned out to be accurate, and it usually is, we could run 1.5 million users with the same 500 load engines. We are pretty confident we could increase the testcase density (the I/O requirements) by at least 2x and the load engines would still have CPU, memory and bandwidth to spare. We have yet to try more than 500 load engines. I’d like to, but since marketing was satisfied with these results, we stopped there.

So our answer to the question is: Load Tester can 5 run 1 million user load tests under the conditions we used. See Lane’s post on running a million-user test for some advice on running big tests.

I prefer to be conservative, so I have to add that your results will be different. The capacity of a single load engine is determined by many factors – the number of testcases/pages/transactions in the testcase, the size of the pages, the time between the pages, etc. Very dense testcases (more txn/sec) will require more bandwidth. Larger testcases (more pages and transactions) will require more memory for processing responses and storing metrics. More complex testcases (with lots of dynamic fields) will require more CPU (though CPU is rarely the limiting factor).

With this release, running massive load tests is easier than it has ever been before. Still, running tests this large is not a trivial task (yet). But if you have a site that can withstand that kind of load, then I’m sure you’re up to the task. Lane has posted some tips that will help you along.

Chris Merrill, Chief Engineer

5 Comments

21 March 2012 ragini

Hi,
Is it possible to do recording through citrix for any other application,as I’m getting a SSL-59 Error during recording through wed performance load tester.Can anyone help me out for this.
Thanks..

21 March 2012 chris

Ragini,

Load Tester is for web applications – so if you want to test non-browser apps, we can’t help you.

I recommend submitting your question to our support team using the support request wizard, which is in Load Tester’s Help menu.

Chris

21 March 2012 ragini

Thanks Chris

17 April 2014 knssub

I am very interested to know how many servers are required for performing such a test. Is the 1million a concurrent users test or is an average load.
Is there any sizing available for sizing typical websites mix of dymaic + static?

17 April 2014 Michael Czeiszperger

The article mentions that 500 computers were used to generate the load. The computing power needed to generate users obviously varies by the test case, but in this case the author mentions that those 500 computers could probably generate 1.5 million users.

The 1 million figure is for concurrent users.

For sizing for a particular dynamic site you’d need to run a test and see how many user a single “load engine” can generate and extrapolate from there. Given that a test case can contain anything, there is no way to predict how much computing power is required to simulate a user without testing.

Add Your Comment

You must be logged in to post a comment.

Resources

Copyright © 2024 Web Performance, Inc.

A Durham web design company

×

(1) 919-845-7601 9AM-5PM EST

Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
How Many Concurrent Users