Troubleshooting the "Too many open files" Error When Keep-Alives Are Used

Document ID : KB000076808
Last Modified Date : 28/05/2018
Show Technical Document Details
Issue:
  • The "Too many open files" error can be solved by KB000042893. While that KB article is just a workaround for the current problem, there is a chance that this could happen in the future. This is relevant if Keep-Alive is used.
Cause:
  • It is a good idea to monitor the number of open files over a period of time to see if they are increasing, and to find the culprit of what is opening so many files. There are two commands below. The first outputs the lsof to '/Desktop/lsof.log' whereas the second command will display the top 20 processes that are opening the most files. This is explained more in this link.
  • lsof > ~/Desktop/lsof.log 
  • cat ~/Desktop/lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
  • After monitoring the open files and you discover that the Gateway is opening many files, it is possible that it is related to "Keep-Alive". This is used in the "Route via HTTP(s)" assertion, within the "HTTP" Tab.
  • When a proxy is configured, the Proxy-Connection: Keep-Alive header gets added by the Gateway. This is done by Apache HttpComponents and HttpClient library. It is not a standard header but widely used by browsers and proxies. This is used because if a client is sending HTTP/1.0 Connection header, some proxies may forward this header blindly to the backend server, which could confuse the server with properties of the client connection and those of the proxy connection. In order to avoid this situation, whenever Client/Gateway know that the request goes through proxy, the Connection header is replaced with Proxy-Connection and hence even if the proxy is blindly forwarding header to the server, it simply ignores it. When Proxy-Connection: Keep-Alive header is present, most proxies send back Proxy-Connection:Close with the response, so that the connection gets closed.
Resolution:
In this situation, the proxy may be failing to send the Proxy-Connection:Close with the response which may trigger the "Too many open files" error logs. When the "Proxy-Connection:Close" header is added manually in the "Route Via HTTP(s)" assertion, the open connection should be closed and the exception disappeared.
Additional Information:
  • CentOS 7 / RHEL 7 - You may not find the lsof command. If not then install it manually by following this link.
  • Related: KB000042893