CA Spectrum SpectroSERVER process cpu spikes and hangs processing events

Document ID : KB000035815
Last Modified Date : 14/02/2018
Show Technical Document Details

Issue:

The CA Spectrum SpectroSERVER process consumes 100% cpu and eventually stops processing updates to Spectrum clients.

 

When reviewing the stack dump files, a similar stack is seen repeatedly in regards to event processing and event procedures:

 

#0 0x00007f3ef1538e00 in CsGNode::rem () from /opt/SPECTRUM/lib/libVPapi.so.1

#1 0x00007f3ef15a40e8 in EventHistory::clearHistory() () from /opt/SPECTRUM/lib/libVPapi.so.1

#2 0x00007f3ef15a4138 in EventHistory::˜EventHistory() () from /opt/SPECTRUM/lib/libVPapi.so.1

#3 0x00007f3ef157975f in CsEventMessage::copy(CsEventMessage const&, int) () from /opt/SPECTRUM/lib/libVPapi.so.1

#4 0x00007f3ef1579ecd in CsEventMessage::CsEventMessage(CsEventMessage const&, int) () from 

/opt/SPECTRUM/lib/libVPapi.so.1

#5 0x00007f3ef5567d9f in EventProcessingQueue::addCurrent(CsModelHandle const&, CsEventMessage const&) () from /opt/SPECTRUM/lib/../SS/libems.so.1

#6 0x00007f3ef5520c09 in CsSMActionCreateEvent::verify_then_create_event(unsigned int, CsModelHandle const&, CsSMEnvironment const*) const () from /opt/SPECTRUM/lib/../SS/libems.so.1

#7 0x00007f3ef5520f9c in CsSMActionCreateEvent::performAction(CsSMEnvironment const*) const () from /opt/SPECTRUM/lib/../SS/libems.so.1

#8 0x00007f3ef551dcd0 in CsSMAction::invoke(CsSMEnvironment const*, Aprisma::ProcedureReturnValue*) const () from /opt/SPECTRUM/lib/../SS/libems.so.1

#9 0x00007f3ef3d697cc in Aprisma::ProcedureBody::invoke(CsVPList const*, Aprisma::ProcedureEnvironment const&) () from /opt/SPECTRUM/lib/../SS/libProcedure.so.1

#10 0x00007f3ef3d749df in Aprisma::ProcedureIf::invoke(Aprisma::Procedure const&, Aprisma::ProcedureEnvironment 

const&) const () from /opt/SPECTRUM/lib/../SS/libProcedure.so.1

#11 0x00007f3ef5512f5c in CsEventDispActionProcedure::process_event(CsEventMessage&, CsGenAttrs const*, int) () from 

/opt/SPECTRUM/lib/../SS/libems.so.1

#12 0x00007f3ef5518ac7 in CsEventDispose::process_event(CsEventMessage&, CsGenAttrs const*, int) () from 

/opt/SPECTRUM/lib/../SS/libems.so.1

#13 0x00007f3ef55095a8 in CsEventDispTable::process_event(CsEventMessage&, CsLscpeHandle const&, CsGenAttrs const*, 

 

Cause:

The issue is due to event loop detection code that the SS uses to detect event loops. 
 

 

Resolution:

CA has created a fix that will limit the loop detection to 100 looped events.  The fix is included in Spectrum_10.00.00.D12 for 10.0 and is included out of the box with CA Spectrum 10.1.  If you need Spectrum_10.00.00.D12 please open a case and request the patch fix.
 

 

Additional Information:  

To change the loop detection value from the default of 100 events, add the following entry to the <SPECROOT>/SS/.vnmrc and provide a value.  Then stop and restart the SpectroSERVER for the change to take effect:

max_event_history_loop_length=