Load Testing: Software or Services?
One of the common questions our customers ask is: “should we buy software and do the load testing ourselves or hire someone to do it?”.
The question is one with few easy answers, but in this blog entry I’ll lead you through the thought process so you can decide which option is the best for your situation.
The first question to ask is how often do you need to do load testing? If you’re planning on testing more than twice a year its usually cost effective to purchase software and learn how to use it. This is true of companies developing their own software where performance is a critical component. Sure, the software is a large up-front cost, but if you compare the per-test prices of owning vs. using an outside service, owning (or long-term leasing) is a fraction of the cost. I’ll talk more about costs of using the software later on.
The second question I ask is “do you have the people with experience to learn how to use the software?”. While we make Load Tester as easy to possible, it still takes some level of expertise to operate. If you either don’t have the people or they’re not available then you have no choice: services is the only option.
The third question is “what are your goals?”. If the goal is to fix a performance problem as quickly as possible, then services are better bet because using experienced performance engineers is the fastest way to find the problem and come up with a solution.
The Software Option
Thinking about buying load testing software and doing it yourself? The biggest thing people overlook when pricing out software vs. services is the very real cost of the employee’s time to learn how to use the software and run the tests, which is often larger than the cost of the software itself. When you have employees who are aren’t experienced performance engineers the majority of the time is spent developing test cases and then trying to figure out the nature of the problems that are reported. Using Load Tester instead of a script-based system saves lots of time, especially in the test development stage, but the more test cases you have and the more complicated they are, the more engineering time the testing process is going to eat up, and the higher the costs.
In the end, buying software is a commitment to dedicating the non-trivial resources needed to meet the performance goals.
The Services Option
A typical testing effort requires someone to configure the tests, run them, do the analysis, and then changes have to be made by a systems administrator, a programmer, a DBA, or a network engineer depending on the nature of the problems found. The plus side of services is you’ve got an engineer on your team with the experience to come up with a test plan and meet your performance goals as soon as possible. The downside is a testing engineer is only one member of the team– you still need the rest of the team to participate in the testing effort. For example, if a change to the web server configuration is called for, your sysadmin will need to do that. If there’s a database issue, then your DBA will need to get involved. The vision of having an outside consulting company come and in and guarantee to fix your problems by themselves is tempting, but highly impractical. Trust me, you don’t want an outside company changing your database tables or your application code; those skills are best kept in-house.
An example was a recent major corporation who was having database problems and wanted their system to scale, and yet there was no DBA on the project. Any development or production team working with a custom database needs access to their own DBA, either full time or on a consultancy basis.
A good analogy is the accounting industry. Each company has their own book keeper and accountant, but to find out if the books are being kept properly you hire an auditor. In the same way, load testing services is like an auditor; we verify that you’re actually getting the performance you’re expecting, but we don’t actually change your ledgers.
3 Comments
23 July 2010 TimGrant
If you use a software to do your testing, then be sure that they inject the proper amount of Network Latency into the Load Test. Otherwise, you give the site an unfar advantage during the test. Also, if the outsouced solution uses a SAAS model from the Internet then be sure that the latency is representative fo your customer traffic. Of course, Network isn’t the only factor in load testing, but it usually gets overlooked.
This article discusses latency a bit more:
http://www.shunra.com/shunrablog/index.php/2010/07/09/load-testing-for-special-events-and-holidays/
10 August 2010 Michael Czeiszperger
That’s true, but it doesn’t have a lot to do with the software vs. services issue, as companies can generate load from inside or outside their networks regardless of who’s doing the testing.
4 March 2011 TimGrant
Thanks for the comment. My point was simply that both software and servcie approaches should properly emulate a propler distribution of real world network conditions into the respective load test framework.