A customer recently changed their networks around. After changing the IP addresses in the appropriate configuration files and verifying that Firewall NATs were in place to allow correct ports to communicate between the tunnel client hub and the tunnel server proxy hub, the tunnel client cannot establish a tunnel connection. Have deleted and recreated the tunnel cert and reapplied on the tunnel client hub. but still cannot get it to connect.
The hub.log file from the tunnel client shows the client is trying to establish a tunnel connection to the tunnel server (on configured tunnel server NAT IP address XXX.XXX.XXX.XXX):
Jul 27 11:51:40:372  hub: SSL handshake start from XXX.XXX.XXX.XXX/48003: before/connect initialization
Jul 27 11:51:40:372  hub: SSL state (connect): before/connect initialization
Jul 27 11:51:40:372  hub: SSL state (connect): SSLv3 write client hello A
But, the SSL connection is failing with an SSL_accept error (5) on the tunnel server side (from the configured tunnel client NAT address YYY.YYY.YYY.YYY):
Jul 27 11:51:46:451  hub: SSL handshake start from YYY.YYY.YYY.YYY/57294: before/accept initialization
Jul 27 11:51:46:451  hub: SSL state (accept): before/accept initialization
Jul 27 11:51:46:451  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: YYY.YYY.YYY.YYY
Usually when we see SSL error 5 in the hub logs it is because there is a security device on one of the networks that is doing a deep inspection or changing the SSL traffic in some way.
Basically the hub is saying the SSL packet is not valid and it has been tampered with. These errors a very hard to track down. They require engaging the network teams on both side to analyze the traffic and see what is tampering with the packets and then putting a rule in place to prevent it.
This behavior is very characteristic of firewall interference and does not appear to be anything related to configuration of the hubs themselves. Something in between these two hubs appears to be blocking the SSL handshake.
The network teams must get involved to see what is tampering with the SSL packets exchanged between the tunnel client and the tunnel server.
Here is a short list of things to check on the firewall:
- Make sure the port 48003 is open for all TCP connections and is not limited to HTTPS or SSL connections. Because we use a proprietary, SSL-based protocol, many firewalls will shut down our connections if the firewall is set to only allow HTTPS or SSL, because the firewall does not recognize our protocol as being HTTPS or SSL specifically. All TCP communication for all protocols should be open on 48003.
- Make sure 'SSL Decryption' option is not enabled on the firewall.
- Make sure 'Deep Packet Inspection' or 'Stateful Packet Inspection' options are not enabled on the firewall.
- Check whether any SSL protocols have been restricted (sslv3, TLS1, etc) as the hub may be using one or more of these protocols.
- Check the firewall logs in general for evidence of dropped/blocked communications between these hubs for other reasons.