Menu

How to get a Quote for Load Testing Services

One of the first questions from customers interested in finding out the performance of their website is “how much will it cost?”.  The main cost drivers for load testing are the number and complexity of the test cases, and the number and size (in concurrent users) of the test iterations.  This blog post will explain how to describe your requirements and get an accurate cost estimate split into two types of charges:  setup, and test iteration.

Setup Costs

The results of a load test are only as good as the quality of the test cases, but there is a large variation in preparation time, and thus cost between the most simple websites and the most complex. Some customers may only be testing a landing-page with a single form, and that would take only a couple of hours to configure. On the other hand, other websites may provide web access to very complex operations software to manage a business, (Peoplesoft for example) and it could take weeks to define and code dozens of lengthy test cases. Most businesses fall somewhere in-between those two extremes.

Server Architecture

Load testing isn’t just about throwing users at a site, you’ll want to know why a site is slow.  We’ll need to setup monitoring on all of the servers involved in the test, and then analyze the data to locate the bottlenecks.  The best way to see at a glance how your system is configured is to provide a system’s diagram showing all of the operating systems and processes involved and how they talk to each other.  If you don’t have this handy, a simple description will do, i.e. “two web servers connect to a single mySQL database”.

Use Cases

In determining a fixed price for services then, its critical to identify exactly what needs to be in each test case. Typically though, we don’t start with test cases, but rather use cases. You can think of use cases as stories told from the user’s perspective about why they are on a website and what they are doing. Web performance is all about delivering the best user experience, so its no wonder we want to start with the user’s perspective.

The first step is to identify what users will be doing on your site in very general terms. For an example let’s first consider what users would want to do at an online shoe store:

  1. Search for types of shoes and view product info
  2. Place a pair of shoes in the shopping cart and make a purchase
  3. Check delivery status
  4. Return a pair of shoes
  5. Sign up for mailing list

While we’re always happy to run as many tests as you want, load testing is fundamentally different from functional testing. In functional testing you want to make sure every single feature of the site works, but with load testing you want to make sure that the site’s bread and butter functions are as fast as possible. In the above list as a first cut we’d recommend only testing use cases #1 and #2, since that’s likely to be 95% of the site’s traffic. Once those are working perfectly, the chances are everything else on the site will also be fast.  Of course, once the main functions are fast, its always possible to go back and add in the other use cases just to be sure.

Once you’ve identified the basic user goals on the system, the next step is to flesh out the description with enough detail so a test engineer can code a test case. Here’s an example of the use case #1, above.

Use Case #1: Casual Shoe Browsing

Use Case #2: Purchasing Shoes

When turning the use cases into test cases the user behavior will be randomized so that each user looks at different types of shoes, each user purchases a different product, and each has a different name and CC info. We fully realize that real users have complex behavior, and that some spend hours looking at shoes, and some look at a pair or two and make an instant purchase. By simulating the typical customer, the above use cases would give us a pretty good picture of the site’s performance, and that is a good time/cost tradeoff most of the time. The next level of accuracy could be obtained by understanding more about your customers and how they behave. For example, say through analytics you determine that men look at an average of 2 pairs of shoes before making a purchase, while women look at an average of 20 pairs of shoes. This information could be used to split the Purchase Shoes use case into two, one for each type of user.

Non-Standard Technology

The fastest and cheapest websites to test are those that are 100% HTML5 compliant.  It takes extra time and care to handle things like ActiveX controls, Java applets, custom browser plugins, etc. Please let us know if your website contains anything other than straight HTML.

Testcase Iteration Charges

What is a Test Iteration?

Once the test cases are written the next step is Phase 4 on the diagram below, otherwise referred to as a “test iteration”. The term iteration is used because the process is usually repeated at least twice. The first test of a site often uncovers an opportunity for making it faster, and you’ll want to make the changes we suggest and re-test the site again to make sure the performance improved.

Note that the Run Load Simulation step below consists of running all of the test cases in parallel, simulating the complex behavior of actual website load.  While one person is searching for a type of shoe, another person may be looking at the details of a particular product.  All of the simulated users are doing their own thing in parallel, just like in real life, and balanced for the popularity of each use case.  For example, if an e-commerce site had a 2% conversion rate, during the test only 2% of the virtual users would be completing the checkout process to make a sale.

services-workflow

How Many Iterations?

The number of iterations you’d want to do depends on where you are in the software lifecycle, and the critical nature of the application. Typically during the development phase, when code is being written, many tests are run as the code changes rapidly. On the other end of the spectrum would be an existing site that’s been working for a while, and running a quarterly test to verify that the peak performance hasn’t degraded. In the former case you might want to run a test iteration a week or more, while in the later perhaps a single iteration would be sufficient.

Another factors include the experience level of the team, the complexity of the applications, and the age of the code.

How Many Concurrent Users?

The cost of an iteration is largely determined by the number of concurrent users.  In this case “concurrent” means users on the site at the same time, wether they’re typing into a form, or just reading a page.  This is not the same as visitors per day or per hour!  If you don’t know the number of concurrent users try this handy load testing calculator.

If calculating the number of concurrent users is impossible, we can alternatively simply find the maximum capacity of your site given the total number of servers, including web servers, database servers, and application servers.

What Time of Day?

While we prefer to work during office hours, Monday through Friday 8AM-6PM EST, sometimes testing must take place outside of those times.  If you need to test after hours just let us know so the additional cost can be worked into the quote.

Total Costs

The combined cost of a services engagement is the one-time setup charge plus the cost of running X number of test iterations.  For example, if the setup charge was $2,000, and a test iteration cost $1,000, the combined price for executing a test iteration twice would be $4,000.

Summary

Putting everything together, the 3 things we need to generate a load testing/web performance services quote are:

  1. Description of the use cases.
  2. A general idea of the number of virtual users
  3. A description of the number and purpose of the servers to be involved in the test. A systems diagram is best, but you can describe the servers in english if none is available.

Add Your Comment

You must be logged in to post a comment.

Resources

Copyright © 2019 Web Performance, Inc.

A Durham web design company

×

(1) 919-845-7601 9AM-5PM EST

Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
What you Need