Remote Hubs lost connection and hub processes are using up to 100% CPU

Document ID : KB000072454
Last Modified Date : 01/03/2018
Show Technical Document Details
Tunnelled hubs connected to 2 secondary hubs that work as hub concentrators for remote hubs have disappeared from IM or showing red.
The hub process on those remote hubs are using 100% CPU (or even more 200% CPU)

Logs are showing the below:

Primary Hub:
Line 89190: Feb 20 12:27:12:640 [8592] hub: TSESS-A-199-39 [prnxxxxxrv44hub] tunnel heartbeat send failed (closing)

Secondary hubs:
Line 207: Feb 20 12:47:21:208 [140289575769856] hub: ssl_server_wait - SSL_accept error (5) on new SSL connection:
Line 215: Feb 20 12:47:21:219 [140289575769856] hub: ssl_server_wait - SSL_accept error (5) on new SSL connection:
Line 223: Feb 20 12:47:21:229 [140289575769856] hub: ssl_server_wait - SSL_accept error (5) on new SSL connection:

Secondary hubs controller:
Feb 20 13:31:19:389 [140503425898240] Controller: SSL - SSL_accept error (5) on new SSL connection
UIM 8.5.1
These sympthomps point to a Network and/or Firewall issue. Firewall configuration was preventing hub to hub communication. If communication is blocked by a Firewall rule or configuration set, tunnels will try to reconnect in a loop, causing the hub processes on the local hub server to use more and more CPU incrementally. This issue has been seen with Cisco ASA Firewall.
•  Double check if all ports are open by checking the UIM port reference
•  Involve the Network Team to verify if any change has been done recently at a network level or to the Firewalls between the involved hubs
•  If needed, restore the current configuration of the Firewall even if no change was performed. (This solved a similar issue with a Cisco ASA Firewall)
•  As soon as communication is restablished completely the hubs will show up immediately and CPU use of the hub processes will drop to 0

Additional Information:
Hub connection issues

Firewall Considerations for UIM Hub Tunnels