SpectroSERVER hanging during startup

Document ID : KB000006806
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

The SpectroSERVER is hanging during startup. A pstack (Linux) or procdump (Windows) of the SpectroSERVER process shows a lot of activity processing events for a South Bound Gateway (SBG). The following is an example of the pstack output:

Thread 1 (Thread 0x2ae922d00a20 (LWP 7933)): 

#0 0x00002ae91e94a0e0 in CsMKHashTable::get_node_value(CsMKHashNode*) const () from /opt/SPECTRUM/lib/libGlobl.so.1 
#1 0x00002ae917698852 in CsLandscape::get_mkd_list(CsAttrDesc const*, CsFindSpec const*, CsMemKeyDesc const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#2 0x00002ae91769d0b3 in CsLandscape::find_keyed_models(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, CsULHashTable*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#3 0x00002ae9176a0d7f in CsLandscape::terminal_find_model_handles(CsFindSpec const*, CsULHashTable*, CsFindSpec::LogicalOp_e, CsError::CsError_e*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#4 0x00002ae9176a0f56 in CsLandscape::find_model_handles(CsFindSpec const*, CsError::CsError_e*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#5 0x00002ae9176a17f6 in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#6 0x00002ae9176a18e1 in CsLandscape::find_models(CsFindSpec const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#7 0x00002ae918751908 in CsVnmLscpeHandle::find_models(CsFindSpec*) const () from /opt/SPECTRUM/lib/../SS/libIHapi.so.1 
#8 0x00002ae91c984a3b in CsIHEventAdmin::findExistingEventModel(CsModelHandle const&, CsBuffer const*, InternetAddress const*) () from /opt/SPECTRUM/lib/../SS/libgag.so.1 
#9 0x00002ae91c990e9d in CsIHEventAdmin::do_trig_event(CsModelHandle const&, CsEventMessage const*) () from /opt/SPECTRUM/lib/../SS/libgag.so.1 
#10 0x00002ae91c993aa5 in CsIHEventAdmin::trig_event(CsModelHandle const&, CsEventMessage const*) () from /opt/SPECTRUM/lib/../SS/libgag.so.1 
#11 0x00002ae9176b28fb in CsModelType::trig_event(CsModelHandle const&, CsEventMessage*, int, CsGenAttrs const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#12 0x00002ae9176cc635 in CsModel::process_event(CsEventMessage*, int, CsGenAttrs const*, CsSecurityIf const*, int) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#13 0x00002ae91bf5f114 in CsDModelIf::process_event(CsEventMessage*, int, int, CsGenAttrs const*) const () from /opt/SPECTRUM/lib/libgserv.so.1 
#14 0x00002ae91a50f9d9 in EventCreationNode::process_node() const () from /opt/SPECTRUM/lib/../SS/libems.so.1 
#15 0x00002ae91a50f91b in EventCreationQueue::process_queued_work(CsWorkNode const*) () from /opt/SPECTRUM/lib/../SS/libems.so.1 
#16 0x00002ae91c1e7092 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1 
#17 0x00002ae91c1e63aa in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1

Cause:

Spectrum is configured with a SBG integration. In the EventAdmin model for this integration, there are thousands of EventModel models. When processing an event for this SBG, Spectrum has to check if there are any existing EventModel models for this event before processing the event even further. When there are thousands of existing EventModel models, this causes Spectrum to hang while it is searching for EventModel models.

Resolution:

Destroy any unneeded EventModel models from within the EventAdmin model.

Additional Information:

Reference http://linuxcommand.org/man_pages/pstack1.html for information on running pstack on Linux

Reference https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx for information on running procdump on Windows.