Menu

Measuring Web Page Load Times using JMeter

At last count there were over 65 separate commercial load testing tools out there, but few with the name recognition of the open source program JMeter. Often people will call us up and ask to compare Load Tester with JMeter, but I only had a cursory look at it many years ago, and couldn’t speak from recent first-hand knowledge. So, when someone called me last week asking about JMeter, it seemed like a good opportunity to give it another look.

Web Performance Consulting

To compare Load Tester and JMeter my first instinct was to record a simple test case with a browser and then run a ramping load, watching how web page load times changed as the load increases. Using the JMeter recorder I grouped together URL requests into pages, and recorded load times at various load levels. While the Load Tester test took about 15 minutes configure and run, it took several days of research to figure out how to set this up in JMeter. Besides not being familiar with JMeter, there’s no automation, so everything had to be configured manually, for example, setting up remote load generators on EC2 alone took two days.

Once the test was configured, the next challenge was to get consistent results. Web pages that should have been loading in the 2 second range were measured by JMeter as loading in 200-400 seconds, which was easy to double check by accessing the system with real desktop browsers during the test. The numbers were off by a couple of orders of magnitude, but it isn’t fair to criticize JMeter for doing what it doesn’t claim to do. Its common knowledge that JMeter doesn’t actually simulate a browser, but I felt like it was worth mentioning again here since so many people who’ve contacted me assume that it does, or that it doesn’t matter.

JMeter and Load Tester have something in common: they are both written in Java. When I was first writing Load Tester it quickly became apparent that the default Java I/O libraries were barely adequate for load testing. The team at Web Performance has spent many man-years writing a custom I/O library optimized for load testing, including millisecond timing accuracy, the efficiency to simulate thousands of users on a single computer, and the most accurate virtual browser available, that simulates every aspect of a browser except for render time. If accurate web page load times are a priority, there’s simply no way to compare these two programs.

There’s also a separate way to measure web pages load times in JMeter, an option to automatically load all of the resources on a page, but that’s quite different from the load generated from a browser. Never-the-less, I came up with a simple test case consisting of a typical uncached web page, a simple ajax page, and a list of articles on a WordPress blog. There was a one second delay between each page, and a ramping load from 0 to 125 concurrent users. Unfortunately it was very difficult to compare because JMeter measured different values on each test. To illustrate, here’s the results of Load Tester running this test, but at the 50 concurrent user level:

Load Tester Web Page Load Time Consistency

Compare that fairly tight grouping with the results of the exact same test run with JMeter, below. I was hoping to come with specific reasons why the web page load times in the two pieces of software were different, but was stymied by the inconsistent results. Note that the test case parameters were identical, the site under test was identical, the load was generated from the same computer, and there was more than enough bandwidth available.

JMeter Web Page Load Time Consistency

One advantage of open source software is the community, so I checked to see how other people were solving this problem. The most common recommendation I found for measuring web page load times with JMeter was to simply run the tool to generate load, and then browse the site with a real browser with a speed-measuring browser plugin. That kinda works, but a sampling approach like that will give you inconsistent measurements. The reason? Performance isn’t constant even if load level is constant. Web page load time at a single load level is best represented in a histogram, with some users getting fast load times, some getting slow load times, spread across a range. Unless you record all of the load times, you’ll never get an accurate picture. Add in a ramping load to the mix, and manually measuring a loaded system with desktop browsers will give you random results within the range at each load level. If all you need to do is a ballpark check to see if the site will fall over under load, then this suffices, but if you’re trying to tune a website, inconsistent measurements aren’t helpful.

The focus of this blog post was on web page load times, because in web performance that’s what the user sees, but what JMeter does seem to do well is measure the performance of individual HTTP requests. If you’re a programmer tuning a back-end call, having JMeter call it over and over would be useful, since you can measure performance in terms of requests processed per second. JMeter also supports a wide range of non-HTTP protocols that Load Tester does not, so if you need to generate SMTP traffic, for example, Load Tester isn’t an option.

While I’m sure that JMeter would get faster for me if I used it more often, the fact is that when you use JMeter you’re trading cost for efficiency. Web Performance has spent years speeding up Load Tester so that our customers can get their job done as quickly as possible, in every area from configuring test cases to special reports. When you buy a commercial product such as Load Tester the money you spend is measured against the increase in productivity and capabilities.

3 Comments

15 May 2013 Does Jmeter give inconsistent results? | Performance Testers Diary

[…] inappropriately. I was there for interested in the following blog post by Michael Czeiszperger (http://www.webperformance.com/load-testing-tools/blog/2013/05/measuring-web-page-load-times-using-jmeter/). Admittedly Michael is the driving force behind WebPerformance a company that has its own […]

15 May 2013 Michael Czeiszperger

Your results were pretty much like mine, that JMeter works consistently for individual URL requests, but not for pages. If all you need are requests for HTML pages without their content, then its fine.

22 July 2013 A Smattering of Selenium #154 | Official Selenium Blog

[…] Measuring Web Page Load Times using JMeter — and read the response articles too for clarification / elaboration […]

Add Your Comment

You must be logged in to post a comment.

Resources

Copyright © 2019 Web Performance, Inc.

A Durham web design company

×

(1) 919-845-7601 9AM-5PM EST

Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
What you Need