Tunnelled hubs connected to 2 secondary hubs that work as hub concentrators for remote hubs have disappeared from IM or showing red.
The hub process on those remote hubs are using 100% CPU (or even more 200% CPU)
Logs are showing the below:
Line 89190: Feb 20 12:27:12:640  hub: TSESS-A-199-39 [prnxxxxxrv44hub] tunnel heartbeat send failed (closing)
Line 207: Feb 20 12:47:21:208  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 93.117.000.202
Line 215: Feb 20 12:47:21:219  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 188.000.17.194
Line 223: Feb 20 12:47:21:229  hub: ssl_server_wait - SSL_accept error (5) on new SSL connection: 185.000.158.84
Secondary hubs controller:
Feb 20 13:31:19:389  Controller: SSL - SSL_accept error (5) on new SSL connection
These sympthomps point to a Network and/or Firewall issue. Firewall configuration was preventing hub to hub communication. If communication is blocked by a Firewall rule or configuration set, tunnels will try to reconnect in a loop, causing the hub processes on the local hub server to use more and more CPU incrementally. This issue has been seen with Cisco ASA Firewall.
Hub connection issues
Firewall Considerations for UIM Hub Tunnels