Transient SpectroSERVER crashes with little or no debug output

Document ID : KB000049950
Last Modified Date : 14/02/2018
Show Technical Document Details

Description:

The SpectroSERVER crashes sporadically and infrequently. The $SPECROOT/SS/log* files might grow by chunks larger than 10MB at a time, and Spectrum Support might see "Exception in CsDbLandscapeRep::store_val" in a SpectroSERVER stack dump. This has been seen on all operating systems and any 9.x.x version of Spectrum.

The cause is an unused statistical logging portion of the Spectrum modeling database is indirectly associated with at least some of these crashes.

Solution:

Verify that the log.* files located in the <SPECROOT>/SS directory are growing by chunks larger than 10MB. Have Spectrum Support confirm the exception seen in the stack dump from a crash (please open a new support issue and upload the <SPECROOT>/SS/support/SpectroSERVER.mdmp file on Windows or the <SPECROOT>/SS/core file on UNIX from the crash).

Once verified, if at Spectrum 9.2.1 or higher add the following line to the $SPECROOT/SS/.vnmrc file:

database_transactions=false

Restart the SpectroSERVER for the .vnmrc file change to take effect. If not at Spectrum 9.2.1 or higher, it will be recommended that you first upgrade to 9.2.0 and 9.2.1 or H06 and higher to get the database_transactions functionality before any additional troubleshooting is done.

For further analysis of the crash, CA Spectrum Support may need the following additional data:

Performance logging output around the time of when the SpectroSERVER crashed. This performance data is automatically logged on each SpectroSERVER system. The following steps should be followed to collect the performance data and additional data needed to start troubleshooting this issue. All data collected can be uploaded and attached to the new support issue;

  • Call Spectrum Support for a copy of the PerfCollector9 performance collection tool.

  • Get an event report for the Universe model for 30 minutes or so, right up to the time of the crash.

  • Do a recursive long listing of the SS directory (from bash);

    ls -alR $SPECROOT/SS > SS_files.out

  • Zip the core or dump file(s) from the $SPECROOT/SS/support directory.

  • Save the Spectrum modeling database via the Spectrum Control Panel or via Online Backup.

  • Operating system performance logs would also be helpful. For Windows:



    1. Click Start, and then click Administrative Tools, and then double-click Event Viewer.

      Figure 1

    2. Select Windows Logs, then Application Logs.

    3. Right-click Application Logs and select "Save Events As." Save as type .evtx.

      For Solaris, /var/adm/messages*
      For Linux, /var/log/messages*