Tunnelled hubs connected to 2 secondary hubs that work as hub concentrators for remote hubs have disappeared from IM or showing red.
The hub process on those remote hubs are using 100% CPU (or even more 200% CPU)
Logs are showing the below:
Line 89190: Feb 20 12:27:12:640  hub: TSESS-A-199-39 [prnxxxxxrv44hub] tunnel heartbeat send failed (closing)
Line 207: Feb 20 12:47:21:208  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 93.117.000.202
Line 215: Feb 20 12:47:21:219  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 188.000.17.194
Line 223: Feb 20 12:47:21:229  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 185.000.158.84
Secondary hubs controller:
Feb 20 13:31:19:389  Controller: SSL - SSL_accept error (5) on new SSL connection
These sympthomps point to a Network and/or Firewall issue. Firewall configuration was preventing hub to hub communication. If communication is blocked by a Firewall rule or configuration set, tunnels will try to reconnect in a loop, causing the hub processes on the local hub server to use more and more CPU incrementally. This issue has been seen with Cisco ASA Firewall.
• Double check if all ports are open by checking the UIM port reference
• Involve the Network Team to verify if any change has been done recently at a network level or to the Firewalls between the involved hubs
• If needed, restore the current configuration of the Firewall even if no change was performed. (This solved a similar issue with a Cisco ASA Firewall)
• As soon as communication is restablished completely the hubs will show up immediately and CPU use of the hub processes will drop to 0
Hub connection issues
Firewall Considerations for UIM Hub Tunnels