Why are my Spectrum servers crashing?

Document ID : KB000106254
Last Modified Date : 13/07/2018
Show Technical Document Details
Introduction:
4 out of 10 SpectroSERVERs running 10.2.3 crashed with a core.
Restoring the previous day's database did not fix the problem.

The following was seen in the vnm.out of the 4 servers
Jul 10 12:29:23 ERROR TRACE at CsWatchNotificationQueue.cc(271): A(n) association notification has exceeded the maximum internal queue size: The client spadmin@sp-oc-02 may be unresponsive. 

A trace of the core showed the following:

Thread 1 (Thread 0x7fd1197f7740 (LWP 2042)): 
#0 0x00007fd1170ebbbc in AccessGroupIH::generate_access_change_event(CsModelHandle const&, CsBuffer*) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#1 0x00007fd1170ebead in AccessGroupIH::process_access_group_change_event(CsModelHandle const&, CsModelHandle const&, CsMTypeHandle const&) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#2 0x00007fd1170ebfa0 in AccessGroupIH::handle_access_group_update(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#3 0x00007fd11906bec9 in CsIHMdlAttrWatcher::notify(CsIncidentParm const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#4 0x00007fd1190de3bf in CsModel::react_to_change(CsVPList&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#5 0x00007fd1190e7618 in CsModel::process_changes() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#6 0x00007fd1191041e1 in CsNotificationMgr::process_queued_work(CsWorkNode const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#7 0x00007fd114314282 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1 
#8 0x00007fd11431358a in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1 
#9 0x00007fd118d6ea63 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1 
#10 0x00007fd10d557d40 in ?? () from /lib64/libc.so.6 


gdbWhere.out shows 

Core was generated by `/opt/spectrum/SS/SpectroSERVER'. 
Program terminated with signal 11, Segmentation fault. 
#0 0x00007fd1170ebbbc in AccessGroupIH::generate_access_change_event(CsModelHandle const&, CsBuffer*) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#0 0x00007fd1170ebbbc in AccessGroupIH::generate_access_change_event(CsModelHandle const&, CsBuffer*) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#1 0x00007fd1170ebead in AccessGroupIH::process_access_group_change_event(CsModelHandle const&, CsModelHandle const&, CsMTypeHandle const&) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#2 0x00007fd1170ebfa0 in AccessGroupIH::handle_access_group_update(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/libmdlsvint.so.1 
#3 0x00007fd11906bec9 in CsIHMdlAttrWatcher::notify(CsIncidentParm const*, CsSecurityIf const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#4 0x00007fd1190de3bf in CsModel::react_to_change(CsVPList&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#5 0x00007fd1190e7618 in CsModel::process_changes() () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#6 0x00007fd1191041e1 in CsNotificationMgr::process_queued_work(CsWorkNode const*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 
#7 0x00007fd114314282 in CsWorkQueue::process_work_node(CsWorkNode*) () from /opt/SPECTRUM/lib/libwkmgr.so.1 
#8 0x00007fd11431358a in CsWorkScheduler::do_work() () from /opt/SPECTRUM/lib/libwkmgr.so.1 
#9 0x00007fd118d6ea63 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1 
#10 0x00007fd10d557d40 in ?? () from /lib64/libc.so.6 
#11 0x0000000000000000 in ?? ()
Question:
Why are my Spectrum servers crashing running version 10.2.3?
Environment:
Spectrum 10.2.3
This was seen originally on a Linux version.
Answer:
10.02.03.D399 has been created to resolve this.

REQUIREMENTS
SPECTRUM 10.02.00,10.02.03 and BMP_10.2.301 are the prerequisites for installation of this patch.

This patch needs to be installed on all the SpectroSERVERs in the same distributed or fault tolerant system.

OVERVIEW
This is a debug patch for the following issue:
SpectroSERVER crashes when access group relation is added/removed for the user.
(DE374460, 01136902)

BILL OF MATERIALS
ReleaseNote.D399.txt
           
runme.exe (Windows)
uninstall_D399.exe (Windows)          
libmdlsvint.dll  (Windows)
libmdlsvutil.dll  (Windows)

runme (Solaris/Linux)
uninstall_D399 (Solaris/Linux)        
libmdlsvint.so.1  (Solaris/Linux)
libmdlsvutil.so.1  (Solaris/Linux)
Additional Information:
Please contact support to get a copy of this patch if you experience similar problems.