Test Only as Accurately as Needed
In previous posts, I’ve talked a lot about the dangers of not accurately simulating the real-world usage. But there is another side to this story: obsessing too much over an accurate simulation can be very costly. There is one very important thing to understand about load testing: No matter how hard we strive to make tests accurate, they are still only an estimate.
Your load test will never match even one single hour of production usage, not even once, no matter how hard you try. Bold statement? Perhaps. Consider this: even if you could capture the exact usage of real-world customers of your site for an hour and simulate it perfectly, it would still be only an estimate, because every other hour of real-world usage will be different. In some ways that is depressing because it means we can never achieve perfection.
But if you embrace this fact and accept that a really good test is, at best, a close approximation of reality, it can be make you a much more productive tester. This can give you flexibility in substituting simpler testcases for more complex ones, as long as the traffic pattern, en masse, resembles the real-world pattern. You can forget that 10,000 real-world users will be coming from 10,000 IP addresses but your test rig can only generate traffic from 100 IP addresses. Accept that everyone tests this way and move on.
By reducing the hours we spend obsessing over getting each testcase perfect, we have more hours we can direct towards running more tests, running them more frequently, with a wider variety of testcases and hitting more critical scenarios. We can test more broadly and thereby decrease the risk to the project. And that is what is really needed!
Chris Merrill, Chief Engineer @ Web Performance