Spectrum crashed and try to start SpectroSERVER shows error "database was not gracefully closed"

Document ID : KB000113595
Last Modified Date : 11/09/2018
Show Technical Document Details
Issue:
We recently experienced a SpectroSERVER crash in CA Spectrum. We tried to start the database back up but now we are seeing an error in the Spectrum Control Panel: 

"ERROR: This application has been started using a Spectrum database which was not gracefully closed. 
The application will now shut down to allow you to load and initialize a previously-saved good database."

How do we get our server back up and running? 
Environment:
Spectrum 9.2 H09 and up
Cause:
In Spectrum 9.2 H09 we introduced a feature to prevent crashed and corrupt databases from causing future issues. If a crashed database is attempted to start, Spectrum will halt the database startup and present the error. 
Resolution:
This feature is meant to prevent corruption related issues. The appropriate course of action is to restore to a last known good backup from before the time of the crash. There are multiple ways to accomplish this: 

NOTE: Before running ANY of these steps, be sure to log into the system as the user that owns the Spectrum installation


1. Restore from Spectrum Control Panel

This can be done by opening the Spectrum Control panel and selecting the Restore Database button. When it asks you to initialize the database, select yes, then select a db save file from before the crash. Once complete, start the SpectroSERVER.

2. Restore from command line
  • cd to <SPECROOT>/SS
  • run  ../SS-Tools/SSdbload -il ../SS-DB-Backup/<your_backup_filename>
  • Once complete, start the SpectroSERVER from Spectrum Control Panel
  • or start the Server from command line: cd to <SPECROOT>/bin and run ./StartSS.pl
NOTE: SSdbload and SSdbsave utility are in SS-Tools directory but MUST be run from /SS directory to work properly. See Additional Information section below for more info on SSdbload and SSdbsave commands. 

If the above steps fail to load a good backup, there may be issue with a corrupt Database Catalog. If this happens, you may need to load legacy.SSdb first, to first load an initial (empty) database, and specify Catalog only. Then load models on top from backup SSdb. 

3. Restore legacy database (initial database) and load models on top
  • cd to <SPECROOT>/SS
  • run  ../SS-Tools/SSdbload -c legacy.SSdb
  • run ../SS-Tools/SSdbload -m ../SS-DB-Backup/<your_backup_filename>
  • Once complete, start the SpectroSERVER
NOTE: Starting in Spectrum 10.2, the legacy.SSdb is a copy of the database as it was prior to any upgrades. Previous to 10.2, the legacy.SSdb is a completely empty database. Starting from 10.2, legacy.SSdb is a database which contains Catalog and Models from the previous Spectrum version (if it was upgraded to the current version, as opposed to a new install). This is to allow a database save to exist immediately after an upgrade completes, as a failsafe should the upgrade fail or some other issue occur before the server can be started and backup after the upgrade. Thus, if loading a 10.2 and higher legacy.SSdb with ../SS-Tools/SSdbload -il legacy.SSdb, it will not appear "empty" but instead will contain models as they existed at the time of the upgrade. Loading legacy.SSdb Catalog Only will appear "empty". 

4. In some extreme cases is may be necessary to completely remove the database files prior to loading a backup.

NOTE: before getting this far, do try the above options (1-3) first. 
  • The SSdb files exist in the $SPECROOT/SS directory as ss*.db (data files) , ss*.ix (index file) , and *.log (log files)
  • Manually delete the files then restore the database using one of the above steps (1-3) 

5. If all else fails, please open a Support Ticket for assistance. Be sure to include the SpectroSERVER Logs for review: 
  • cd to <SPECROOT>
  • run .//bin/support/getSpectrumInfo.sh lite
  • upload <SPECROOT>/logs_<hostname>_<timestamp>.tar.gz to the case

NOTE: If you have also experienced a crash on Archive Manager DDM database, see this article for instructions on repairing Archive Manager: 

https://comm.support.ca.com/kb/spectrum-archive-manager-not-starting-after-system-crash/kb000057378

 
Additional Information:

More information on SSdbload & SSdbsave commands: https://docops.ca.com/ca-spectrum/10-3-0/en/administrating/database-management/spectroserver-database-maintenance/ca-spectrum-database-tools/ssdbload/

Open a Support Ticket here: https://support.ca.com/us.html