Remote Hubs lost connection and hub processes are using up to 100% CPU

Document ID : KB000072454
Last Modified Date : 01/03/2018
Show Technical Document Details
Issue:
Tunnelled hubs connected to 2 secondary hubs that work as hub concentrators for remote hubs have disappeared from IM or showing red.
The hub process on those remote hubs are using 100% CPU (or even more 200% CPU)

Logs are showing the below:

Primary Hub:
Line 89190: Feb 20 12:27:12:640 [8592] hub: TSESS-A-199-39 [prnxxxxxrv44hub] tunnel heartbeat send failed (closing)

Secondary hubs:
Line 207: Feb 20 12:47:21:208 [140289575769856] hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 93.117.000.202
Line 215: Feb 20 12:47:21:219 [140289575769856] hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 188.000.17.194
Line 223: Feb 20 12:47:21:229 [140289575769856] hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 185.000.158.84

Secondary hubs controller:
Feb 20 13:31:19:389 [140503425898240] Controller: SSL - SSL_accept error (5) on new SSL connection
Environment:
UIM 8.5.1
 
Cause:
These sympthomps point to a Network and/or Firewall issue. Firewall configuration was preventing hub to hub communication. If communication is blocked by a Firewall rule or configuration set, tunnels will try to reconnect in a loop, causing the hub processes on the local hub server to use more and more CPU incrementally. This issue has been seen with Cisco ASA Firewall.
Resolution:
•  Double check if all ports are open by checking the UIM port reference
•  Involve the Network Team to verify if any change has been done recently at a network level or to the Firewalls between the involved hubs
•  If needed, restore the current configuration of the Firewall even if no change was performed. (This solved a similar issue with a Cisco ASA Firewall)
•  As soon as communication is restablished completely the hubs will show up immediately and CPU use of the hub processes will drop to 0

 
Additional Information:
Hub connection issues
https://comm.support.ca.com/kb/hub-connection-issues/KB000044046

Firewall Considerations for UIM Hub Tunnels
https://comm.support.ca.com/kb/Firewall-Considerations-for-UIM-Hub-Tunnels/KB000010564