Trap Director considerations for CA Spectrum 9.2 and 9.3

Document ID : KB000003883
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

Trap Director was enhanced in 9.2.3 H11 to allow for more capacity:

Cause:

Symptom: Historically, there have been two main problem areas with Trap Director functionality:1) Trap Director may not be able to forward traps to remote servers as fast as they are received. This can result in the Trap Director SpectroSERVER (SS) terminating due memory exhaustion.2) The Trap Director does not immediately recognize when models are moved from one SpectroSERVER to another. These changes aren't accounted for until either the Trap Director cache ages out or the Trap Director SpectroSERVER is restarted.

Resolution:

The following enhancements have been made to Trap Director functionality.1) Trap forwarding to remote landscapes is now done in an asynchronous manner. This can greatly increase forwarding throughput in a geographically distributed environment.2) Trap Director cache accuracy has been improved. Remote SpectroSERVERs now notify the Trap Director SpectroSERVER of events that require the cache to be updated.3) A new Trap Management sub-view has been added to the VNM model Information tab, which contains control options and informational fields related to trap processing.

(H11, 168540) 
(H11, 110598, 21048835-1) 
(H11, 170858, 21258291-1)

As part of this work, there are two areas of concern:

  1. The SpectroSERVER may crash when forwarding traps 

  2. When the Trap Director SpectroSERVER is started in a distributed environment with mixed Spectrum patch versions (for example: some servers running H10 and some running H12), the SpectroSERVERs may crash.

A stack dump similar to the following may be observed (please contact CA Spectrum Support for analysis of crash files to obtain the stack info):

#0 0xf70d4ef0 in CsAlertFwdMgr::rem_pim_from_cache () from 
/opt/SPECTRUM/lib/../SS/libalert.so.1 
#1 0xf70da8b0 in CsAlertFwdMgr::clear_central_cache_contents () from 
/opt/SPECTRUM/lib/../SS/libalert.so.1 
#2 0xf70da96c in CsAlertFwdMgr::connect_callback () from 
 /opt/SPECTRUM/lib/../SS/libalert.so.1 
#3 0xf6d6e4c1 in LandscapeConnectionMonitor::doNotificationLoop () from 
/opt/SPECTRUM/lib/libgserv.so.1 
#4 0xf6d6e517 in LandscapeConnectionMonitor::notificationWrapper () from 
/opt/SPECTRUM/lib/libgserv.so.1 
#5 0xf7cdadfb in Thread::call_thread_function () from 
/opt/SPECTRUM/lib/libmoot.so.1 
#6 0xf7cdc5a7 in thread_start () from /opt/SPECTRUM/lib/libmoot.so.1 
#7 0x005bba1b in makecontext () from /lib/libc.so.6 
#8 0x0cfbf148 in ?? () 
#9 0x00000000 in ?? ()

Solution:

Please read the entire solutions below in regards to Concern #1 and Concern #2

The crash for concern #1 is resolved in:

Spectrum H12 (Spectrum_09.02.03.H12) and greater and in Spectrum 9.3 and greater.

  • Symptom: 

    1. Intermittent SpectroSERVER crash when updating Trap Director cache due upon connection establishment. 

    2. SpectroSERVER memory growth due to Trap Director forward attempts to unconnected landscapes.

    3. Secondary SpectroSERVER memory growth due to combination of hot-standby and trap director.

  • Resolution:

    1. Resolved crash when updating Trap Director cache.

    2. Resolved several reasons for memory growth when Trap Director is enabled.

    3. Disabled trap forwarding on hot-standby secondary SpectroSERVER.

    (H12, 206791, 21397144-1) 
    (H12, 203987, 21391315-1)

For concern #2:

If you are installing H12 (or greater) or 9.3 (or greater) in a distributed SpectroSERVER environment that is currently running a Spectrum release prior to H12, then it is HIGHLY recommended that you disable Trap Director functionality until ALL SpectroSERVERs have the H12 (or greater) or 9.3 (or greater) installs completed across all SpectroSERVERs. For example, if you are currently running Spectrum 9.2.3 (H10) and you will be installing H12, disable the Trap Director functionality on the VNM model under the Trap Management view. Once ALL of your SpectroSERVERs have the patch installed (even fault tolerant SpectroSERVERs ), then you can turn the Trap Director back on. If you run H12 or 9.3 in a mixed environment running Spectrum version H11 or prior, with Trap Director enabled, there is potential for SpectroSERVERs running Trap Director to crash.

Once you have all SpectroSERVERS running H12 (or greater) or 9.3 (or greater) you do not need to disable the Trap Director functionality prior to any future installs.