CA Spectrum SS crashes in trap forwarding code when there is a disconnect between SpectroSERVERs and there are duplicate IP addresses involved.

Document ID : KB000006531
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

The SpectroSERVER may crash when trap forwarding is enabled and the following .vnmrc entry is in use:

alert_forwarding_connect_full_flush=false

This entry disables the rebuilding of the IPAddress cache for SpectroSERVERs that have network connectivity issues.

If the above are true, when there is a disconnect between the Main Location Server and a distributed SpectroSERVER the SS may crash when a trap is received for an IPAddress known by duplicate models. The stack analysis will show the following stack where the actions rem_pim_from_cache and clear_central_cache_contents are seen from the CsAlertFwdMgr function:

 

(gdb) where 

#0 0x00007f4a80197ca7 in CsPIMDesc::hash(CsPIMDesc const*) () 

from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 

#1 0x00007f4a78d08278 in CsGHashTable::get_node(void const*) const () 

from /opt/SPECTRUM/lib/libGlobl.so.1 

#2 0x00007f4a78d082e1 in CsGHashTable::get(void const*) () 

from /opt/SPECTRUM/lib/libGlobl.so.1 

#3 0x00007f4a7cfaaa76 in CsAlertFwdMgr::rem_pim_from_cache(CsTrapStormDetector*) () 

from /opt/SPECTRUM/lib/../SS/libalert.so.1 

#4 0x00007f4a7cfab35b in CsAlertFwdMgr::clear_central_cache_contents(CsLscpeHandle const&) () from /opt/SPECTRUM/lib/../SS/libalert.so.1 

#5 0x00007f4a7cfab467 in CsAlertFwdMgr::connect_callback(void*, CsError::CsError_e, ConnectionChangeInfo const&) () from /opt/SPECTRUM/lib/../SS/libalert.so.1 

#6 0x00007f4a7b7a9b4b in LandscapeConnectionMonitor::doNotificationLoop() const () 

from /opt/SPECTRUM/lib/libgserv.so.1 

#7 0x00007f4a7fdf9843 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1 

#8 0x0000003e73a438b0 in ?? () from /lib64/libc.so.6 

#9 0x0000000000000000 in ?? ()

Cause:

There was a code flaw in the way the IP address cache was being handled when there were models with the same IP address.

Resolution:

This is tentatively scheduled to be fixed in CA Spectrum release 10.3 due out in late 2017.  For Spectrum releases 10.2.0 and 10.2.1 the resolution is to install the following patch on all SpectroSERVERs:

10.2.0 - Spectrum_10.02.00.PTF_10.2.040 

10.2.1 - Spectrum_10.02.01.PTF_10.2.112 

If you need either patch please open a case withe CA Support online and request the patch.

 

The release notes show the following:

Symptom: Rare SpectroSERVER crash when Trap Director is enabled and 

        alert_forwarding_connect_full_flush=false and landscape with duplicate 

        IP models disconnects and reconnects.

 

Resolution: Resolved SpectroSERVER crash due to memory corruption.

(DE284928, 00668193)