Load Testing Back to Basics: Avoiding the KeepAliveTimeout Race Condition
You’ve recorded your test case, configured your datasets, and run your replays. You start up the load test and … you see numerous errors like this:
“The connection with the server was unexpectedly closed before starting the response.”
What’s going on? Well, one common reason for this error is a connection-related race condition between Load Tester and the web server due to the server’s configured persistent connection timeout.
Persistent connections are an HTTP mechanism for minimizing network connection overhead between the browser and the web server. If the client has the Connection request header set to Keep-Alive, and the server responds with the Keep-Alive server header, the connections will be maintained to allow the browser to pipeline through more requests at a later time. However, the web server cannot allow the browser to monopolize these connections forever. After a certain point, the server is permitted to cut the connection; normally, this occurs after a certain period of browser inactivity. This timeout is controlled in Apache 2.2 via the KeepAliveTimeout directive, which defaults to five seconds, and in IIS 7 via the connectionTimeout option under the “limits” directive, which defaults to two minutes.
The race condition is created when the page think time in Load Tester is set to a value that overlaps the KeepAliveTimeout or connectionTimeout setting: for example, when KeepAliveTimeout is set to ten seconds and the page think time is set to ten seconds. When the think time expires, the load engine begins to pipeline additional requests through the open connections. However, at the same time, the server is shutting down the connection. If the request arrives in time, the server will respond to it, and reset the persistent connection timeout. However, if the request does not arrive in time, the server finishes closing the connection and sends a TCP FIN packet to the client to finalize the connection tear-down. From the perspective of the load engine, the server has closed a connection that has a request in progress, and that’s why Load Tester reports an error.
This sort of problem occurs routinely in a browser, but is transparent to the user; the browser simply opens a new connection, repeats the request, and moves on. However, Load Tester has no way to know if this behavior is normal or not – other conditions such as firewall connection timeouts can also generate this situation – so we must report the error.
In Load Tester you can also create this problem in a load configuration by setting the think time variation range to a value that causes some think times to overlap the persistent connection timeout. For example, if your think time is ten seconds and the KeepAliveTimeout or connectionTimeout is set to twelve seconds, setting the think time range value to 75-150% will overlap the timeout. Thus, the subset of pages with think times that fall around twelve seconds will see the connection errors as requests fail to make it to the server before the connection is closed.
To avoid these errors, you should set KeepAliveTimeout or connectionTimeout to a value that is outside the think time range. If you feel that you must test with the timeout values as-is, you should alter your think times and/or user count to retain the same load levels while avoiding the problem. Given typical internet latencies, offsetting the think time even by one second either way can minimize the connection errors.
Happy Testing!
Matt Drew
Web Performance Test Engineer