The CA SpectroServer crashes when creating "mini" dump files for SS performance issues.

Document ID : KB000045415
Last Modified Date : 14/02/2018
Show Technical Document Details

Issue:

The SpectroSERVER crashes when trying to create "mini" dump files for SS performance issues.  The internal code to Spectrum will attempt to create a stack dump of the SpectroSERVER process for troubleshooting purposes.  When that process fails the SpectroSERVER may crash.  The stack from the core file will show "createminidump" along with CORBA calls as can be seen here (please contact CA Support for analysis of the core file):

 

Core was generated by `/spectrum/SS/SpectroSERVER'. 

Program terminated with signal 11, Segmentation fault. 

#0 0xf6833e0a in MiniDump::createMiniDump() () from /opt/SPECTRUM/lib/libGlobl.so.1 

(gdb) info stack 

#0 0xf6833e0a in MiniDump::createMiniDump() () from /opt/SPECTRUM/lib/libGlobl.so.1 

#1 0xf7755da1 in MakeMootRequest_ptr(CsCorbaSrvcItcM*, int, unsigned long, ...) () 

from /opt/SPECTRUM/lib/libitc.so.1 

#2 0xf7d5ba5c in CModelDomainItcN::startWatchAttrValsOfModelsByIDs(CORBA_ULongSeq const&, CORBA_ULongSeq const&, CsCAttribute::CsCAttrReadMode_eList const&, CsCAttrValWatchCB*) () 

from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1 

#3 0xf7483128 in POA_CsCModelDomain::_startWatchAttrValsOfModelsByIDs(void*, CORBA_MarshalInBuffer&, char const*, VISReplyHandler&) () from /opt/SPECTRUM/lib/libssorb.so.1 

#4 0xf50e05d5 in VISSkelInfo::execute(VISGIOPRequest const&, CORBA_MarshalInBuffer&, VISReplyHandler&) () 

from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#5 0xf50f328a in VISGIOPProtocolAdapter::_do_request(VISGIOPRequest*, VISGlobalTable*) () 

from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#6 0xf50f2070 in VISGIOPProtocolAdapter::dispatchMessage(ProtocolEngine::Message*, VISThreadSpecificData*) 

() from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#7 0xf51e9b19 in VISThreadPoolDispatcher::do_work(ProtocolEngine::ProtocolAdapter*, long&, VISThreadSpecificData*) () from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#8 0xf51e9db9 in VISDispatchWorker::do_work(VISThreadSpecificData*) () 

from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#9 0xf51dd553 in VISThreadPool::_perform_task(VISTPoolWorker*) () from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#10 0xf51dd9af in VISTPoolWorker::begin() () from /opt/SPECTRUM/lib/liborbcore_r.so.8.0 

#11 0xf4e97200 in VISThread::_start(void*) () from /opt/SPECTRUM/lib/libvport_r.so 

#12 0x0041f832 in start_thread () from /lib/libpthread.so.0 

#13 0x0037f0ae in clone () from /lib/libc.so.6 

 

Resolution:

If you are running CA Spectrum 9.4.x this issue is due to a stack overflow that is fixed in 10.x.  

If yo are running CA Spectrum 10.x this issue is under review and will be fixed in a future release of CA Spectrum.

In the meantime, to prevent the SpectroSERVER from crashing, please add the following entry to the <SPECROOT>/SS/.vnmrc file and restart the SpectroSERVER:

enable_itc_moot_perf_debug=FALSE

Please note that this does disable automatic performance stack generation, so if you are experiencing an SS cpu issue, you will need to manually generate stack dumps with pstack for linux and procdump for Windows.