UIM alarms stopped appearing in Spectrum. Tried the following but none of these helped resolve this issue:
- Stopped the spectrumgtw and ems probes, acknowledge all UIM alarms and restarted the probes
- On the Spectrum OneClick server. stopped and started tomcat
- Executed the validate connect test from the spectrumgtw probe in the Admin Console, and it came back with valid connection
No errors in the ems or spectrumgtw logs to identify why this is occurring.
UIM Sever 8.61
ems 9.03 upgraded to 10.17
nas 8.56HF5 upgraded to 9.03
Spectrum UIM Service 8.65
One oracle probe was publishing 22K+ informational alarms every monitoring interval. This was flooding the nas causing the nisqueue.db file to grow in size and actually identified a defect in the 9.03 nas release.
Modify the oracle probe configuration to stop publishing 22K+ unnecessary informational alarms.
Development recommended the following to detect this condition in the future:
- Monitor the nas nisqueue.db file and if the size of the file continues to grow over time to raise an alarm (potential alarm storm in progress). The dirscan probe was suggested as a means of monitoring for this condition.
- Configure storm protection in the robot spooler probes to prevent message storms from ever reaching the nas probe. The drawback to this issue, is that you cannot conifgure this on hubs that have monitoring probes that might generate message storms.
- Configure the nas probe and enable storm protection. This option allows users to choose between Suppression-id, Robot, or Probe as the source of the message filter. and quarantine messages should a storm occur in the future. The drawback to this is that the nas still has to process all 22K+ alarms in the alarm flood to determine if they need to be quarantined or not.