10.3 SS crashes during AutoDiscovery

Document ID : KB000123149
Last Modified Date : 18/12/2018
Show Technical Document Details
Issue:
When running discoveries on very large devices or running discoveries that cover thousands of devices, the SpectroSERVER may crash.

The following stack shows from the crash file with the problem being CsOIDValueList::unpack:

#0 0x00007f23b01b7bad in __memmove_ssse3_back () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64 elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7_4.2.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 nss-softokn-freebl-3.28.3-8.el7_4.x86_64 systemd-libs-219-42.el7_4.7.x86_64 xz-libs-5.2.2-1.el7.x86_64
(gdb) where
#0 0x00007f23b01b7bad in __memmove_ssse3_back () from /lib64/libc.so.6
#1 0x00007f23b46dd434 in CsBuffer::operator new(unsigned long, unsigned char const*, unsigned char const**) ()
from /opt/SPECTRUM/lib/libGlobl.so.1
#2 0x00007f23b4c06610 in attrunpack(CsAttrDesc::CsAttrType_e, unsigned char const*, unsigned char const**) ()
from /opt/SPECTRUM/lib/libVPapi.so.1
#3 0x00007f23b4c53693 in CsOIDValue::CsOIDValue(CsAttrDesc::CsAttrType_e, unsigned char const*, unsigned char const**, unsigned char)
() from /opt/SPECTRUM/lib/libVPapi.so.1
#4 0x00007f23b4c544fa in CsOIDValueList::unpack(unsigned char const*, unsigned char const**) () from /opt/SPECTRUM/lib/libVPapi.so.1
#5 0x00007f23b4c5458d in CsOIDValueList::CsOIDValueList(CsBuffer const&) () from /opt/SPECTRUM/lib/libVPapi.so.1
#6 0x00007f23bbb6ab5c in CsModel::gather_list(CsMTypeAttr const*, CsObjectID*, CsObjectID*, unsigned int, int, CsError::CsError_e*)
() from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#7 0x00007f23bbb6b80d in CsModel::read_list(CsAttrReadReqList*, CsAttrValRspList*, CsDcmArgs const*) ()
from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#8 0x00007f23bbb6ef65 in CsModel::read(CsAttrReadReqList*, CsDcmArgs const*, CsSecurityIf const*) ()
from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#9 0x00007f23b706707d in CsDModelIf::read(CsAttrReadReqList*, CsSecurityIf const*) const () from 
Cause:
When the AutoDiscovery completes, it creates the AdiscResultSet model which shows the history of the discovery.  The problem is that the attribute can only hold 4MB worth of data.  If you discover thousands of devices, or hundreds of large devices (lots of interfaces) the data stored is larger than 4MB which would cause the SpectroSERVER to crash.
Resolution:
There are couple of ways to resolve this:
1.  Break up your discovery so that you are not discovering as much in one discovery.
If you are discovering 10000 devices in 1 discovery break it up into 4 so you are only discovering 2500 per discovery.
2.  Install 10.3 with the 2nd BMP (Spectrum_10.03.00.BMP_10.3.002) and  install Spectrum_10.03.00.D19 on top.
In Spectrum_10.03.00.D19 the code was changed so that not only will the SS not crash, but Spectrum does not store interfaces that do not have a mac address associated.  Interface information is not available in the History anyway, so there was no need to save it.  
3. Upgrade to Spectrum 10.3.1 when it becomes available (first quarter FY 19) where the code changes from both Spectrum_10.03.00.BMP_10.3.002 and Spectrum_10.03.00.D19 are included.
Additional Information:
Even though the interfaces with no mac address are not stored, when the modeling portion runs, it will still model those interfaces.