Local Network Connections
We recently had a customer that exhibited a particular error message whenever the site went past 60-70 simultaneous users. There were two servers under test, a webserver and a database/application server. The only symptom we could see in the load test reports was a large number of failed TCP connections on the webserver, but not on the database server.
Their webserver was using CGI to build and display pages. The CGI script would make a local TCP connection to a communications service running on the same machine, which would then seek out a remote application server to service the request. At about 60-70 simultaneous users, we would start receiving error messages from the server indicating that the communications service was unable to connect to the application server.
It turns out that on Windows, there is a maximum number of TCP ports that a user can use, and the default is 5000. This particular application was making local connections to a service that provided communications to the application/database server, and then that service would make multiple outgoing connections to the remote application server. These connections quickly exceeded the maximum available ports that the user could use, and started generating TCP connection failures.
The immediate solution was to raise the MaxUserPort limit to the maximum allowable (65534), but the design itself is limited, and clearly the site would be permanently limited to about x8 the current load, or about 480-560 users.
Network problems are not always on the network. 🙂