A primary SpectroSERVER experienced a performance issue, resulting in the SpectroSERVER being unresponsive. As the process itself was still running, the secondary didn't take over, so the whole landscape was unavailable (red). The SpectroSERVER had to be stopped with kill -9.
It is not clear how long this situation already persisted before we noticed it, as we rely on monitoring the SpectroSERVER process to identify such issues. the main clue are these repeated messages in the VNM.OUT:
WARNING at ItcRequestParmsCorba.cc(92): CORBA request not accepted after 120000ms.
We have the following .vnmrc parameter in place: itc_request_timeout=120000