Menu

Load Engine Tuning: JVM Memory Optimization

The Web Performance load engine is the software Load Tester uses to create virtual users and generate load on the target.  As with Load Tester, the load engine is a Java-based application that runs on its own Java virtual machine, which is included in the installation.  There are two places the load engine is used: the local engine, which is included with Load Tester and runs inside the Load Tester JVM; and the remote engine, which is a standalone installation with its own JVM.  The local engine is limited and intended mainly for replays and small tests, so in this article we’ll be exclusively discussing the remote engine.

There are two main configuration options that are critical to understanding how to tune the Load Engine for maximum performance: the JVM heap size and the memory limits of your operating system.  The JVM heap size is the memory reserved by the JVM for use by the running Java code, and is configured when the JVM is started.  A Java program running inside the JVM can only see and use the heap memory.  On 32-bit systems, the theoretical maximum heap size is 4GB; however, because of the virtual memory limits described below, the practical maximum heap on 32-bit operating systems is usually around 1.4 to 1.6GB due to the JVM itself needing additional memory outside the heap for its own purposes.  Critically, for this discussion, one of those purposes is allocating memory to support each thread running inside the JVM.  This means that each Java thread not only uses heap space while running, but also increases the JVM overhead.

The memory limits of your operating system are how much memory your application can “see” and use.  The limits are a function primarily of the hardware platform your operating system runs on, although some operating systems offer a few choices about how to configure virtual memory.  For our purposes, the virtual or physical memory limit controls the total amount of memory that can be used by the Java process, which includes the heap size and the overhead required by the JVM.  On 32-bit systems with a 32-bit JVM this virtual memory limit is usually 2GB  or 3GB (Linux systems with the kernel option CONFIG_HIGHMEM4G or CONFIG_HIGHMEM64G set).  On 64-bit systems with a 64-bit JVM the virtual memory limits are OS dependent and usually so high as to be effectively unlimited, so the important limit becomes the amount of available physical memory the JVM can access.

Also, note that both the maximum JVM heap size and the virtual memory limits can be reduced by various factors – for example, an operating system running inside a virtual machine environment on a 32-bit host may be subject to either virtual memory limitations on the host, or “physical” limits imposed by the VM configuration.  This may reduce the amount of memory that can be allocated per process inside the VM, which in turn will reduce the amount of heap memory that can be allocated in the JVM.

For our load engine, the heap size is configured at JVM start time by two options in the Load_Engine.lax config file:

lax.nl.java.option.java.heap.size.initial=200M
lax.nl.java.option.java.heap.size.max=200M

These options correspond to the Java command-line options -Xms and -Xmx, and default to 200MB (the actual Java default is 64MB, which is almost always too small for our purposes).  As with Load Tester, the 200MB default is usually not enough for serious testing.  Both the instant load engine bootable CD and our cloud load engines are a bit smarter – they allocate 75% of the available system memory (after OS overhead), subject to the 32-bit virtual memory limits.  This is a good starting point for optimizing the load engine.

The load engine uses a simple virtual user model that assigns one thread to one virtual user.  Thus, every time we add a virtual user, we spawn a new thread inside the JVM.  Each thread uses a portion of the JVM heap for its own variables and buffers.  However, the JVM also uses additional memory outside the heap to support each thread.  This means that there are actually two limits that affect the load engine – the JVM heap size and the total size of the process.

how a Java thread uses memory

how a Java thread uses memory

When the JVM runs out of heap space, the load engine will generally refuse to add more virtual users.  This can manifest itself as a premature halt or slowdown in the user ramp, since that particular load engine will be unable to add additional users.  The memory usage data reported by the load engine is the JVM heap usage, so the reported memory usage on the load engine will approach 100%.  This is the most common type of failure since heap usage per thread is usually much larger than the thread overhead.

However, if you set the JVM heap size too high, it will also cause problems for the JVM.  On 32-bit systems, if the heap size is set so high that the JVM overhead would exceed the virtual memory limit or the operating system cannot allocate enough memory for the heap, the load engine will simply fail to start:

A 32-bit Linux JVM set to use 3500MB of memory fails to start

A 32-bit Linux JVM set to use 3500MB of memory fails to start

The worst case is when the JVM is able to start, but is not able to allocate enough memory for the thread overhead.  Normally, this will cause the load engine to crash or hang.

On 64-bit systems things are somewhat easier, as you don’t have to worry about the virtual memory limits.  However, on a 64-bit system another danger becomes more prominent: invalidating your test by forcing the load engine to swap to disk.  The JVM does not  handle being swapped to disk very well – performance becomes very sluggish and the system will exhibit high CPU usage from the Java process.  This occurs most often with extremely high numbers of users (10,000+), as the thread overhead becomes very large in that case and can quickly exceed the physical memory available.  Also, be aware that approaching these user levels can cause you to run into other operating system limits, such as the number of outgoing TCP ports (65535) or various file-handle or thread limits.

In order to find the optimal heap setting, it is necessary to observe the load engine during a test.  If you see the process memory approach the virtual or physical memory limits, then tune down the heap size.  If you see the load engine approaching 100% heap usage, tune up the heap size and then monitor the process make sure you aren’t approaching the virtual or physical memory limits.  Finally, tests will vary, and there are diminishing returns to constantly tweaking the load engine settings.  If you are in a situation where you find yourself constantly changing these settings, it’s usually quicker and easier to add another load engine and spread the load.  Happy testing!

Matt Drew

Add Your Comment

You must be logged in to post a comment.

Resources

Search

Copyright © 2017 Web Performance, Inc.

A Durham web design company

×

For Prices Call (1) 919-845-7601 9AM-5PM EST

After hours? Prefer email? Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
Product Needs
×

Request a Quote

Our experienced performance engineers have tuned hundreds of systems for companies large and small, and know just where to look, saving you time to market and money. We'll run your website through a complete performance evaluation, then tell you exactly how many users your site can support, including such important details as the effects of "the last mile." We'll also pinpoint potential problem areas and give you a full report detailing what needs to be fixed and where.

About You
Product Needs