Alarm_enrichment queue backed up

Document ID : KB000047856
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue Summary

The alarm_enrichment queue is backing up. At loglevel 5, the error can be found in the logs:

No element found for key level

Example Screenshot of Issue
?
User-added image
Issue
Resolution Steps

Use the Dr. Nimbus tool to fetch unread messages from the alarm_enrichment queue. If the messages are missing any fields other than md5sum, or if they do not match the expected Subject, then those messages could be holding the queue up. To access the DrNimBUS tool, on the primary hub select Start button->All Programs->Nimsoft Monitoring and then Tools->DrNimBUS.

Use this button to use the tool on the primary hub:
User-added image

Choose a queue, and then fetch single messages using the + button.

Considerations:

1) It helps to set the bulk size on nas/ alarm_enrichment down to 10. If the probe still cannot load the first ten messages, then you know one or more of the first ten messages is causing an issue.

2) If the messages contain a subject of QOS_MESSAGE, then they are the incorrect subject and cannot be processed by alarm_enrichment. Fetch all QOS_MESSAGEs out until the subjects are alarm or filter on alarm as a Subject.

3) The DrNimbus tool permanently marks the messages as read. Once you see them in DrNimbus, they cannot be delivered to the probe.

4) Using Dr. Nimbus to read the alarm_enrichment queue may disconnect alarm_enrichment. In this case, the queue will show as yellow, and the probe will need to be restarted.

Resolution Screenshot

?
User-added image