Fail to route from CA APIM gateway to Amazon AWS server

Document ID : KB000112378
Last Modified Date : 31/08/2018
Show Technical Document Details
Issue:
The gateway has a back end server on Amazon.
It's working fine most of the time, but intermittently, the following connection issue could occur,
WARNING 1286 com.l7tech.server.policy.assertion.ServerHttpRoutingAssertion: 4042: Problem routing to <endpoint url>. Error msg: Unable to obtain HTTP response from <endpoint url>: Connect to <host:port> timed out.

When the problem occurred, the backend AWS instance looks good, and running curl command to access the backend url has no problem.
 
Cause:
The root cause is Amazon AWS server uses dynamic IP by default.
And gateway has DNS caching by default. 
Therefore when gateway try to establish connection to the old IP which is not valid anymore, it will get a connection timeout.


 
Resolution:
There could be 3 options.
1. Use Elastic IP for AWS servers.
2. Restart the gateway (each time the problem occurred)
3. Disable DNS caching
Add a new line to /opt/SecureSpan/Gateway/node/default/etc/conf/node.properties file,
node.java.opts = -Dsun.net.inetaddr.ttl=0 

Restart the gateway to take effect

Resolution 1 is recommended.
Additional Information:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html

Thinking beyond this case,
The route via http(s) assertion has keep-alive option enabled by default. If Amazon closes the connection without notifying gateway, it could cause another type of connection issue.