Spectrum Fault Isolation not suppressing or asserting alarms as expected.

Document ID : KB000036118
Last Modified Date : 20/09/2018
Show Technical Document Details
Issue:

During a major outage, we did not receive an expected critical alarm on a model. No alarm was asserted on the model as expected. Spectrum Fault Isolation not suppressing or asserting alarms as expected.

In the following screen shot, the circled router model was down and should have alarmed Critical with the "DEVICE HAS STOPPED RESPONDING TO POLLS" alarm. However, Spectrum did not assert a Critical alarm. The model stayed green but could not be contacted by Spectrum.

195739_1.png

 

 

Environment:
Spectrum 9.x
Spectrum 10.x
 
Cause:

The 0x10d35 event associated with the "DEVICE HAS STOPPED RESPONDING TO POLLS" alarm was modified as follows:

0x00010d35 R CA.EventPair, 0x10d30, "0xfff0000e -:-", 600 R CA.EventCombo, "0xfff00030 -:-", 300, "0x10d30 -:-"

Out of the box, the 0x10d35 event is defined in the $SPECROOT/SS/CsVendor/Cabletron/EventDisp file as follows:

0x00010d35 E 75 A 3, 0x00010009,N

Resolution:

Modifying the out of the box events may have undesirable results. The underlying Fault Isolation, Impact Analysis and Fault Suppression code is looking for specific events with specific conditions and specific probable cause codes in order to function properly. Changing any one of these from the default may have undesirable results.

In the above scenario, after changing the 0x10d35 event back to the out of the box definition and replicating the outage, Spectrum alarmed as expected:

195739_2.png

Additional Information:
Here is a list of events which we do not recommend to customize. Check this list and compare to the event as seen in Event Configuration Editor. 

https://docops.ca.com/ca-spectrum/10-2-3/en/managing-network/event-configuration/event-and-alarm-customization

If an Out of Box event has been modified, it will show as Author=Custom. In order to remove customizations you can either edit the <SPECROOT>/custom/Events/eventdisp file and remove the entry for the event from this file, or you can Delete the event from Event Configuration. After Deleting a customtized OOB event and refreshing the Events list in ECE, the Author should return to Author=CA.

You then want to do one of two things to reload the changes into the events cache: 

1. Reload event configuration via VNM model settings:
- on VNM model go to SpectroSERVER Control subview
- click "Update Event Configuration"

or

2. Restart the SpectroSERVER