{"id":3099,"date":"2012-03-19T16:09:06","date_gmt":"2012-03-19T20:09:06","guid":{"rendered":"http:\/\/www.webperformance.com\/load-testing-tools\/blog\/?p=3099"},"modified":"2014-07-14T16:21:32","modified_gmt":"2014-07-14T20:21:32","slug":"howto-generate-1-million-virtual-users-with-load-tester-5-0-pro","status":"publish","type":"post","link":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/2012\/03\/howto-generate-1-million-virtual-users-with-load-tester-5-0-pro\/","title":{"rendered":"HowTo: Generate 1 Million Virtual Users with Load Tester 5.0 PRO"},"content":{"rendered":"

Leading up to the release of Load Tester 5.0, the Web Performance development team focused heavily on improving our capability to run massive load tests.\u00a0 Today, Load Tester 5.0 is specifically engineered to deliver as many as 1 million virtual users while controlling 500 remote load engines.<\/p>\n

This is a “how to” article for Load Tester 5.0 users wanting to run their own massive load tests.<\/p>\n

\"Graph<\/a><\/p>\n

Before you Start<\/h3>\n

There are a few things you absolutely need.\u00a0 First and foremost is a modern workstation for the controller.\u00a0 By modern, I mean a 64-bit architecture with at least 7 GB of working memory.\u00a0 This is an absolute must.\u00a0 To benefit from the 64-bit architecture, it will be necessary to run the 64-bit version of the controller as well.\u00a0 CPU power is a less critical:\u00a0 We ran successful tests with an in-house Intel i7-860 and an Amazon EC2 High-CPU Extra Large Instance (c1.xlarge).\u00a0 The controller can take advantage of up to three hardware threads.<\/p>\n

You may also save time during the engine-initialization phase by having plenty of upload bandwidth, especially if you have a testcase that uploads a lot of data or if you have a very large dataset that is marked both re-usable and sharable.\u00a0 Most testcases don’t fit this description, but if this does describe you, consider installing Load Tester on an Amazon cloud instance where it will have oodles of bidirectional bandwidth.<\/p>\n

(Remember, these specifications are for those wanting to run load tests to 1 million users — for a typical 1-hour test running between 100 and 10,000 users, system requirements are largely a non-issue.)<\/p>\n

Finally, you’ll need a lot of cloud instances.\u00a0 Load Tester is integrated with Amazon EC2, so you can launch arbitrary numbers of cloud instances from within Load Tester at the push of a button.\u00a0 You’ll need to estimate the number of cloud instances required to successfully execute your test.\u00a0 For example, if you run 10 load engines to 20,000 virtual users before they cap out, then you’ll need a bare minimum of 500 load engines to run to 1 million users.\u00a0 Not all test cases will scale this efficiently, especially those that have very short think times.\u00a0 Load engines self-monitor and will stop adding virtual users when they are at capacity.<\/p>\n

Be aware that Amazon imposes a quota on the maximum number of cloud instances that can be run by any customer at any given time.\u00a0 The default quota at this time is 20 engines per region.\u00a0 It usually takes a few days for the quota increase to be approved, so plan accordingly.\u00a0 (Amazon has a contact form for this specific purpose at: http:\/\/aws.amazon.com\/contact-us\/ec2-request\/<\/a>.)<\/p>\n

This probably goes without saying, but your website will need to be reachable from the Amazon cloud.<\/p>\n

Tuning Web Performance Load Tester for Maximum Performance<\/h3>\n

You’ll need to allocate extra memory to Load Tester, and you’ll need to do this up-front so that Load Tester’s pre-flight self-checks don’t indicate a shortage.\u00a0 Edit the “webperformance.ini” file in Load Tester’s installation directory.\u00a0 The two lines “-Xms1000m” and “-Xmx1000m” (the actual value you see might be different, but the command-line flag is the same) control the size of Load Tester’s memory heap.\u00a0 If you set this number too low, Load Tester will run out of memory.\u00a0 If you set it too high, you might also starve certain processes that live outside of the heap.\u00a0 About 2\/3rds of available hardware memory is a good rule of thumb.\u00a0 For example, on an 8-gigabyte workstation, I would set this to about 5500m.<\/p>\n

In the load configuration, you should disable the options named “Detailed Page Durations” and “Individual URL Metrics.”\u00a0 Also, under “Error Recording”, don’t tune the “Number of descriptions” or “Number of pages” settings above their default values — these are already counted per-engine.\u00a0 These options involve features of Load Tester that don’t scale well to hundreds of thousands of users.\u00a0 “Detailed Page Durations” in particular collects data samples on a per-page-load basis.<\/p>\n

Evaluating the Health of your Load Test<\/h3>\n

Best practice is to consider the failure of even one load engine to invalidate the test results from the moment of failure forward.\u00a0 When testing the high-performance features of Load Tester, we never encountered a load engine failure mid-test; however, as with all engineered systems, increasing the number of components increases the probability of a failure.\u00a0 Evidence of a lost load engine will first manifest when composite live statistics (such as the yellow average page duration chart) stop updating.\u00a0 These statistics are only updated after all load engines have reported.\u00a0 Eventually, either the backlog will clear or Load Tester’s built-in self-checks will determine that a material failure has occurred and report it explicitly.<\/p>\n

Use the Engines View to monitor the health of the load engines as they are running.\u00a0 This view monitors CPU utilization, memory, upstream and downstream bandwidth, and ping time (round-trip latency).\u00a0 Excessive values for any of these numbers can be considered evidence of a problem.<\/p>\n

The Status View monitors the health of the controller.\u00a0 The “Memory” sub-heading of this view can report memory pressure.\u00a0 You can allocate more memory to Load Tester’s heap (using the procedure described above) if this value seems problematic.\u00a0 Using the “Cleanup…” tool before each load test to remove unneeded materials from the repository can also reduce memory pressure.\u00a0 The previous advice, regarding the “Detailed Page Durations” and “Individual URL Metrics” also strongly impact memory utilization.<\/p>\n

The “Diagnostic” sub-heading monitors Load Tester’s two data processing pipelines and will only show activity during massive load tests.\u00a0 The visualization queue handles chart updates, and you can clear any backlog on this queue by disabling unneeded charts.\u00a0 The metric storage queue is a critical function and should not exceed zero, although momentary bursts are acceptable; if this queue is overloaded consider moving the controller to a machine with more CPU and memory.<\/p>\n

Summary of Common Problems, Causes, and Solutions when Running Massive Load Tests<\/h3>\n
    \n
  1. Chart updates are slow or lag behind the actual running time of the test<\/strong>:\u00a0 Caused by displaying too much chart data or temporarily or permanently losing contact with at least one load engine.\u00a0 Consider closing unnecessary charts, especially charts with many elements such as those in the engines view.\u00a0 If you have permanently lost contact with a load engine, the test is no longer valid after that time.<\/li>\n
  2. Load tester reports an “out of memory” condition<\/strong>: Caused by configuring Load Tester’s memory heap too low or too high, or because the load test is gathering more data than can be stored in available memory, or because pre-existing contents of an open repository are eating up memory that would otherwise be used for the load test.\u00a0 Remember that you need to turn off\u00a0“Detailed Page Durations” and “Individual URL Metrics.” Consider using the “Cleanup…” tool to cut down on repository size, and allocate more memory to Load Tester if the memory status indicator indicates a shortage.\u00a0 Don’t expect to run a massive load test using the 32-bit version of Load Tester.<\/li>\n
  3. It takes too long to begin the test<\/strong>: Caused by a combination of large datasets or file uploads in the test case and insufficient upstream bandwidth.\u00a0 Consider installing the controller on the Amazon cloud to net more bandwidth and lower latency to the load engines.<\/li>\n
  4. Load Tester stops adding users<\/strong>:\u00a0 This is a common problem even for small or moderate load tests.\u00a0 You don’t have enough engines to generate the requested load.<\/li>\n
  5. Other problems<\/strong>: If you encounter a problem not listed here, I want to know.\u00a0 Please file a ticket with Web Performance’s support tracker<\/a>.<\/li>\n<\/ol>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    Leading up to the release of Load Tester 5.0, the Web Performance development team focused heavily on improving our capability to run massive load tests.\u00a0 Today, Load Tester 5.0 is specifically engineered to deliver as many as 1 million virtual users while controlling 500 remote load engines.
    \nThis is a “how to” article for Load Tester 5.0 users wanting to run their own massive load tests.
    \n    <\/a>
    \nBefore you Start
    \nThere are a few things you absolutely need.\u00a0 First and foremost is a modern workstation for the controller.\u00a0 By modern, I mean a 64-bit architecture with at least 7 GB of working memory.\u00a0 …     Continue reading »<\/a><\/p>\n","protected":false},"author":53,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,42],"tags":[],"class_list":["post-3099","post","type-post","status-publish","format-standard","hentry","category-load-testing","category-load-tester-software"],"_links":{"self":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/3099","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/users\/53"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/comments?post=3099"}],"version-history":[{"count":12,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/3099\/revisions"}],"predecessor-version":[{"id":4923,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/3099\/revisions\/4923"}],"wp:attachment":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/media?parent=3099"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/categories?post=3099"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/tags?post=3099"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}