Connection lost between the MLS SpectroSERVER and Remote DSS SpectroSERVER

Document ID : KB000006132
Last Modified Date : 05/11/2018
Show Technical Document Details
Issue:

During the time the Online Backup runs on the MLS SpectroSERVER, we see the following two alarms generated for some of our remote landscapes:

"Connections to Landscape on Secondary Server" (event 0x10c1a) 

"Trap(s) Handled Locally Due To Non-Responsive Landscape" (event 0x10dc2) 

 

The alarms remain once connection is re-established with the primary MLS. This has been an on-going issue since Spectrum 9.3, and very sporadic.

Environment:
We are running a Distributed, Fault Tolerant environment for Spectrum.
Cause:

This is a very rare issue, where the MLS and DSS attempt to establish a connection at the exact same time. This causes Spectrum to see duplicate connections, which forces us to close the connections. Due to the timing of this issue, Spectrum tends to get caught in a loop, so the cycle starts all over.

Basically what happens is:

Both sides, at exactly the same time, connect to each other. 

Both sides, at exactly the same time, detect that now there are two connections, and closes one (each sides closes a different connection).

Now there are no connections, so both sides open new connection… 

Repeat.

 

This can be seen by enabling a Communication Trace on each of the SpecrtroSERVERs.

  1. Locating the VNM model in OneClick.
  2. In the Component Detail Pane - Information Tab, scroll down and expand the Dynamic Debugging subview.
  3. Enable the "General SpectroSERVER Communication" debug.
  4. The output will be written to the $SPECROOT/SS/VNM.OUT file.  

When the issue is happening, you should see something like the following, which repeats throughout the log each time a connection attempt is made.

COMM TRACE at CsConnMgr.cc(1613): Starting an accept from NULL at xxx.xx.xx.46, 0x0, handle = 123140, interest = 1 

COMM TRACE at CsConnMgr.cc(1626): Open successful for accept from SpectroSERVER at xxx.xx.xx.46, 0xbeef, handle = 123140, interest = 1 

COMM TRACE at CsConnMgr.cc(1643): Found duplicate opened conn, closing accept from SpectroSERVER at xxx.xx.xx.xx, 0xbeef, handle = 123140, interest = 1

COMM TRACE at CsConnMgr.cc(1659): Closing duplicate opening connection to SpectroSERVER at r4xxxx1002, 0xbeef, handle = 123135, interest = 1 

COMM TRACE at CsConnMgr.cc(1347): Recording a pending connection to NULL at r4xxxx1002, 0xbeef, handle = 123141, interest = 1 

COMM TRACE at CsBaseVConn.cc(170): Version number = 10.1.1.000 is ok for SpectroSERVER at r4xxxx1002, 0xbeef 

COMM TRACE at CsVnmConn.cc(203): Peer connection security passed by SpectroSERVER at r4xxxx1002, 0xbeef 

COMM TRACE at CsVPConnect.cc(586): Request mgr is usable for conn to SpectroSERVER at r4xxxx1002, 0xbeef 

COMM TRACE at CsBaseVConn.cc(315): Synching parm block version to 30 for conn to SpectroSERVER at r4xxxx1002, 0xbeef 

COMM TRACE at CsConnMgr.cc(1429): Returning CsVPConnHandle = 123141 for conn to r4xxxx1002, 0xbeef 

COMM TRACE at CsConnMgr.cc(1613): Starting an accept from NULL at xxx.xx.xx.46, 0x0, handle = 123146, interest = 1 

COMM TRACE at CsConnMgr.cc(1626): Open successful for accept from SpectroSERVER at xxx.xx.xx.46, 0xbeef, handle = 123146, interest = 1 

COMM TRACE at CsConnMgr.cc(1643): Found duplicate opened conn, closing accept from SpectroSERVER at xxx.xx.xx.46, 0xbeef, handle = 123146, interest = 1

COMM TRACE at CsConnMgr.cc(1659): Closing duplicate opening connection to SpectroSERVER at r4xxxx1002, 0xbeef, handle = 123141, interest = 1

Resolution:

This has been resolved in CA Spectrum 10.2.2.

This issue is addressed in Spectrum 10.0 and Spectrum 10.2 with the following patches:

  • For Spectrum 10.0, open a case and request Spectrum_10.00.00.D16.
  • For Spectrum 10.2, open a case and request Spectrum_10.02.00.PTF_10.2.026


If you continue to see this problem, it is due to configuring multiple SpectroSERVERs as Trap Directors.  In a distributed Spectrum environment, only one SpectroSERVER is supported as a Trap Director.

As noted in CA Spectrum documentation you can only designate one SpectroSERVER as the Trap Director:
https://docops.ca.com/ca-spectrum/10-2-3/en/administrating/distributed-spectroserver-administration/working-with-trap-director 

To use Trap Director, designate one SpectroSERVER as the Trap Director server. Enable Trap Director on that server. Then configure that server as the network management station (NMS) recipient for traps from devices that are modeled in the remote landscapes.

Note: Designating more than one SpectroSERVER as a Trap Director in the same distributed SpectroSERVER (DSS) environment is not supported.