{"id":915,"date":"2010-06-22T15:35:28","date_gmt":"2010-06-22T19:35:28","guid":{"rendered":"http:\/\/www.webperformanceinc.com\/load_testing\/blog\/?p=915"},"modified":"2010-07-20T09:31:15","modified_gmt":"2010-07-20T13:31:15","slug":"back-to-basics-tcp-connection-failures","status":"publish","type":"post","link":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/2010\/06\/back-to-basics-tcp-connection-failures\/","title":{"rendered":"Load Testing Back to Basics: TCP Connection Failures"},"content":{"rendered":"

Load Tester is a web site load testing tool, and as such we deal primarily with the most popular Internet communications protocol: the Hypertext Transfer Protocol<\/a>, or HTTP, which controls the request and transmission of web pages between browser clients and web servers.\u00a0 HTTP is based on a lower-level protocol known as the Transmission Control Protocol<\/a>, or TCP.\u00a0 \u00a0 For the most part, TCP works in the background, but its proper function is critical to your website, and problems at the TCP level can show up in many different ways during a load test.\u00a0 These errors can sometimes be difficult to troubleshoot, requiring a packet sniffer such as Wireshark or tcpdump to analyze, while others are simpler.<\/p>\n

TCP uses the concept of “ports” to identify and organize connections.\u00a0 For every TCP connection, there are two ports – the source port, and the destination port.\u00a0 For our purposes, the most important ports are port 80 and port 443, which are the two most common ports utilized by web servers – 80 for normal HTTP traffic, and 443 for SSL-encrypted traffic.\u00a0 A typical TCP connection from a client to a webserver will involve a random source port such as 44567, and a destination port on the server of port 80.\u00a0 Each web server can accept many hundreds of connections on port 80, but each connection must come from a different source port on each client.<\/p>\n

To create these connections between ports, TCP relies on a three-way handshake.\u00a0 The requesting client first sends a packet with the TCP SYN flag set, indicating that it wants to open a connection.\u00a0 If the server has a process listening on the destination port, it will respond with a packet that has both the SYN flag set and the ACK flag set, which acknowledges the client’s SYN and indicates that a connection can be created on that port.\u00a0 The client then sends a packet with the ACK flag set back to the server, and the connection is established.\u00a0 The current connections can be viewed using the netstat tool on both Windows and Linux.<\/p>\n

What does it look like when a TCP connection attempt fails?\u00a0 The TCP packet with the SYN flag is sent from the client, which in our case is a load engine.\u00a0 If the server sees such a packet, but does not have a process listening on the target port, it will typically respond with a TCP packet that has the ACK and RST flags set – a TCP reset.\u00a0 This tells the client that connections are not available on this port.<\/p>\n

\"Load

Load Tester showing a connection refused (ACK RST)<\/p><\/div>\n

This screenshot shows the result of a load engine failing to connect to the server.\u00a0 In this case, you can see that I attempted to connect to TCP port 442, which doesn’t have a web server running on it (or any other service, for that matter).\u00a0 Note that the response was received quickly, in about 1 second, indicating that the remote server saw the ill-fated packet and responded.\u00a0 The most important thing to know about this error is that it is one of the most reliable errors that you’ll see – either the Load Tester controller or the load engine really is having trouble connecting to the site.\u00a0 The most common reason for this is that either the site is down, or there is a firewall that is blocking the load engine but not the controller.<\/p>\n

So … what happens when the remote server does not respond?<\/p>\n

\"Load

Load Tester showing a connection timeout (dropped packet)<\/p><\/div>\n

This screenshot shows the same attempted connection, only this time, no response was received from the target server – not even the TCP reset that indicates connections are not available on the target port.\u00a0 Note how long it takes for Load Tester to report an error – 21 seconds, in this case.\u00a0 I induced this error by configuring the Linux iptables firewall to drop all incoming packets on TCP port 442, so the server’s TCP stack never saw the incoming SYN packet and thus did not respond to it – from the server’s perspective, the packet never arrived.\u00a0 A similar error will occur if the server cannot be reached for some reason; for example if you attempt to connect to the wrong hostname, the server is offline, or your traffic is being misrouted between the client or load engine and the server.\u00a0 If you see these kinds of errors, then the first thing you should do is make sure that the server is up, and that any HTTP proxy servers necessary to reach the server are configured correctly.<\/p>\n

Of course, TCP connections can also fail after<\/em> a connection has been established.\u00a0 Here’s an example:<\/p>\n

\"Load

Load Tester showing a server connection termination<\/p><\/div>\n

This error message is much less clear.\u00a0 Did the server close the connection on purpose?\u00a0 If so, why?\u00a0 If not, what happened? Did the process handling the server connection crash or return bad data?\u00a0 In this case, it’s useful to know what Load Tester considers to be a succesful connection.\u00a0 Load Tester expects there to be HTTP headers, followed by data.\u00a0 In this case, we did not finish receiving the HTTP headers, and so Load Tester considers the connection incomplete.\u00a0 Load Tester failed to receive the headers in this case because I induced this error by attempting to elicit an HTTP response from the Secure Shell (ssh) service listening on TCP port 22, which terminated the connection after receiving what it saw as invalid data – Load Tester’s HTTP request.<\/p>\n

In a real test, there’s a pretty large number of things that can cause this error, from server process crashes or errors, to overly aggressive firewalls, to reverse proxy failures, to misdirected traffic on a load balancer.\u00a0 In such a case, a traffic analyzer such as Wireshark or tcpdump can be very helpful in determining what is happening.\u00a0 Note that you may need to observe traffic in more locations that in front of the load engine or the controller though, as traffic can be altered by firewalls and load balancers.<\/p>\n

Matt Drew<\/p>\n","protected":false},"excerpt":{"rendered":"

Load Tester is a web site load testing tool, and as such we deal primarily with the most popular Internet communications protocol: the Hypertext Transfer Protocol<\/a>, or HTTP, which controls the request and transmission of web pages between browser clients and web servers.\u00a0 HTTP is based on a lower-level protocol known as the Transmission Control Protocol<\/a>, or TCP.\u00a0 \u00a0 For the most part, TCP works in the background, but its proper function is critical to your website, and problems at the TCP level can show up in many different ways during a load test.\u00a0 … Continue reading »<\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[87,88,86],"class_list":["post-915","post","type-post","status-publish","format-standard","hentry","category-load-testing","tag-load-tester-errors","tag-networking-basics","tag-tcp-connection-failures"],"_links":{"self":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/915","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/comments?post=915"}],"version-history":[{"count":11,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/915\/revisions"}],"predecessor-version":[{"id":1203,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/915\/revisions\/1203"}],"wp:attachment":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/media?parent=915"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/categories?post=915"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/tags?post=915"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}