Tools for Troubleshooting Spectrum Tomcat Memory Issues

Document ID : KB000124372
Last Modified Date : 10/01/2019
Show Technical Document Details
Introduction:
At times, the Spectrum Tomcat service may face a memory issue and crash. When this happens, a heapdump (.hprof) file is supposed to be generated to provide information explaining the root cause of the issue. Sometimes this does not happen or the file becomes corrupted. The steps below outline how to manually generate this heapdump, if needed.
Environment:
CA Spectrum 10.2.x
CA Spectrum 10.3.x
Instructions:
NOTES:   If using a Linux server, the DISPLAY variable will need to be set properly in order to launch the GUI for each of the applications below
                 If using a Windows server, the commands below will need to be run from a bash shell. To launch a bash shell, open command prompt and type: bash -login


We can use a tool "JConsole" that is shipped with Spectrum to monitor the process memory in real time. Below are the steps to display this information:

1. First, obtain the Process ID (PID) for the Spectrum Tomcat process.
2. From the shell run the following commands:

cd <SPECROOT>/Java/bin
jconsole <tomcat_PID>

3. Once loaded, you will see a live overview of the tomcat process with graphs for memory usage, threads, classes, etc..


Sometimes the Tomcat process may be facing a memory issue and it will be necessary to generate a heap dump or .hprof file for CA Support to review in order to determine the root cause. Spectrum also ships a tool "jVisualVM" which will allow you to generate these heap dumps manually. Below are the steps to complete this process:

1. Open the Spectrum Tomcat Configuration File. These files differ between Windows and Linux OS so please find the appropriate filename below:

Windows -> <SPECROOT>/tomcat/bin/OneClickService.conf
Linux -> <SPECROOT>/tomcat/bin/catalina.sh

2. Add the following lines to the bottom of the file, then save the file with the changes:

jvm_opt=-Dcom.sun.management.jmxremote.port=8086
jvm_opt=-Dcom.sun.management.jmxremote.ssl=false
jvm_opt=-Dcom.sun.management.jmxremote.authenticate=false

2. Restart the Spectrum tomcat process.
3. From the shell run the following commands:

cd <SPECROOT>/Java/bin
jvisualvm

4. Once loaded, you should see tomcat with the PID next to it under Applications -> Local. Double-click this service.
5. Select the Monitor tab and make sure CPU, Memory, Classes and Threads are all checked.
6. Select the Heap Dump button.
7. Once complete, you should see the [heapdump] under the Tomcat Process in the Applications pane. Please select the [heapdump] then File -> Save As.

Once saved, please send over the .hprof file to Spectrum Product Support for review.