Why does the SpectroSERVER shutdown very soon after startup trying to allocate 11 bytes of memory?

Document ID : KB000107474
Last Modified Date : 20/07/2018
Show Technical Document Details
Introduction:
The problem is seen more often in older versions of Spectrum before Spectrum 10.0 because of the limitation of 32bit memory for the SpectroSERVER process.
However the same problem can be seen in newer versions of Spectrum as well.
Question:
Why does the SpectroSERVER shut itself down very quickly after startup with an error message in the VNM.OUT that it was unable to allocate a very small amount of memory (11 bytes)?  In the VNM.OUT I can see that the SpectroSERVER has printed this stack after shutting down.

Jul 18 13:16:17 ERROR TRACE at VNM.cc(797): S:/CA/SPECTRUM/SS/SpectroSERVER.exe is out of memory while allocating 11 bytes. Scheduling shutdown.

0x113727aa libGlobl.dll!CsSymbolInfo::print_current_stack
0x100aaf59 libsskrnl.dll!SearchManager::process_queued_work
0x11732c5b libPort.dll!Cs_new_handler
0x117364c6 libPort.dll!malloc
0x1136ed82 libGlobl.dll!Cs_strdup
0x7d32f1 libalert.dll!CsSnmpAlert::CsSnmpAlert
0x7d53c2 libalert.dll!TrapQueueNode::TrapQueueNode
0x7d58ea libalert.dll!RemoteForwardingTrapQueue::queue_trap
0x7d628e libalert.dll!CsAlertManagerMT::process_trap_MT
0x7d6676 libalert.dll!CsAlertManagerMT::trap_receiver
0x11ac4cd8 libmoot.dll!Thread::call_thread_function
0x11ac3e76 libmoot.dll!IOEvent::IOEvent
0x7536cbb2 kernel32.dll!CreateFiberEx

Jul 18 13:16:20 ERROR TRACE at VNM.cc(841): C:/SPECTRUM/SS/SpectroSERVER.exe is out of memory while allocating 12 bytes. Continuing shutdown.
CA memory allocation error. Aborting.
Environment:
Spectrum 9.x
Spectrum 10.0
Spectrum 10.1
Spectrum 10.2
Answer:
The cause of this excessive memory consumption in a very short time is due to an excessive number of traps that are being sent to the SpectroSERVER.  Spectrum does have trap storm alarming and suppression built in, but Spectrum as designed, must first read the traps, which uses memory, in order to know that it is part of a trap storm and not a valid trap.  This causes the memory exhaustion and will eventually use up all of the free memory and Spectrum will shutdown as a result.
You will need to locate the devices that are sending an excessive number of traps and resolve the problem that causes them to send the traps to Spectrum.
If it is not known or it is not possible to resolve this problem the only option is to disable all trap processing on the SpectroSERVER.
You can do this by editing the .vnmrc file ($SPECROOT/SS) and changing the option:

from:
snmp_trap_port_enabled
to:
snmp_trap_port_enabled=FALSE

This will allow the SpectroSERVER to start up and it shouldn´t crash as it is not processing traps. 

Alternatively you can also change the SNMP Trap Port number in the .vnmrc file

snmp_trap_port=
to
snmp_trap_port=163

This will also stop Spectrum from receiving traps on the default port of 162.

Once the devices have been identified that are sending excessive traps via wireshark or tcpdump, you can stop the SpectroSERVER and change back these settings to the original values of empty (default).