SpectroSERVER will not finish starting

Document ID : KB000004844
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

SpectroSERVER appears to hang on "Model Activiation" and will not start

Environment:
Spectrum 10.1.1Windows OS
Cause:

There is a known issue with Juniper Slot models in Spectrum 10.0, 10.1 and 10.1.2 - Spectrum will create duplicate slot/chassis models on Juniper Devices, which causes performance issues. Please see the following tech doc: 

TEC1666174Juniper routers chassis reconfiguration (reconfiguring) causing SpectroSERVER cpu problems and duplicate chassis modules in CA Spectrum 10.1.0, 10.1.1, and 10.1.2

It may be likely that if a large Discovery of Juniper models was performed, and the SpectroSERVER required a restart, that the SpectroSERVER's startup may be delayed in the pre model activation stage, accounting for all of the additional Juniper slot models. This may exhibit in a situation where the SpectroSERVER never moves past "Model Activation" stage as shown on the Spectrum Control Panel (VNM.out). Other symptoms include: 

- Check Model Activation Threads in OneClick - shows "0"

- Check Model Activation with VNMSH CLI - shows "0"

Resolution:

This situation is a two part fix. 

First part is to make sure Spectrum is patched with the appropriate version that fixes the Duplicate Juniper Slot model issue as described in TEC1666174.

1. Initialize and reload the SSdb on the SS but do not start the SS

2. On the problematic SpectroSERVERs, stop processd

3. Install the appropriate patch - in this case the issue was seen with Spectrum 10.1.1 so the patch version is 10.1.1 - Spectrum_10.01.01.PTF_10.1.109

4. Start processd but do NOT start the SS

Second part is to run a script on the SpectroSERVER Database (SSdb file) that removes ALL inherited Juniper slot models. Then, assuming Spectrum is patched, running a Reconfigure on the Juniper devices will prevent duplicates and reduce performance issues. 

1. As the Spectrum user, from the command prompt, run >bash -login

2. Nav to <SPECROOT>/SS

4. Place SSdebug in the SS directory

5. Create a new file named "juniper_ssdebug_scrip.sh" and include the script syntax listed below.

6. Run "juniper_ssdebug_script.sh"

7. Verify the script creates the file "attr_change_playfile.out" and that this file increases in size.

NOTE: It is important to note that depending on the number of juniper Devices originally discovered, and the number of duplicate slot models Spectrum may have already created, this process may take a significant amount of time. As long as the attr_change_playfile.out increases in size, the script it still running. 

8. From the Spectrum Control Panel, Start the SpectroSERVER

9. Reconfigure the Juniper devices

 

"juniper_ssdebug_script.sh" SCRIPT SYNTAX: 

<-- start script

SSDEBUG_PATH=`echo $PWD`

 

#get a list of all models in the database

echo "getting list of models from currently loaded database........"

$SSDEBUG_PATH/SSdebug.exe -database -models > models.tmp

 

#-------------------------------------------------------

#get rid of header stuff in the models.tmp file

total_lines=`wc -l models.tmp | awk '{ print $1 }' `

 

lines=`expr $total_lines - 8`

 

#create a SSdebug.exe play file that lists the model handle

 

echo "creating SSdebug.exe play file........"

#for model in `tail -n $lines models.tmp | grep JuniperSlot | awk '{ print $2 } '`

#do

#echo "destroy $model" >> attr_change_playfile

#done

 

tail -n $lines models.tmp | grep JuniperSlot | awk '{ print "destroy",$2 }' > attr_change_playfile

#-------------------------------------------------------

#now run the play file

echo "Deleting JuniperSlot models....."

 

 

$SSDEBUG_PATH/SSdebug.exe -database -play attr_change_playfile attr_change_playfile.out

 

end script --> 

Additional Information:

* Verify version of Spectrum via the .history file, located in <SPECROOT>/Install-Tools

* Regardless of SpectroSERVER Database Startup issues, it is highly recommended to install the appropriate patch for your version of Spectrum 10 - as outlined in TEC1666174

http://www.ca.com/us/services-support/ca-support/ca-support-online/knowledge-base-articles.tec1666174.html

* This Juniper Slot issue is resolved in Spectrum 10.2