High usage of java.exe process at Service Catalog server when integrated to Service Desk Manager

Document ID : KB000006608
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

Sometimes the Service Catalog's Java process will reach 100% CPU utilization.

During these times the PDM CATALOG SYNC DAEMON (pdm_catalog_sync) can be see back in Service Desk Manager (SDM) server showing as in "Connecting" state.

In ServiceCatalog.log and view.log file you could see below messages appearing:

 

INFO | jvm 1 | YYYY/MM/DD HH:MM:SS | java.lang.OutOfMemoryError: GC overhead limit exceeded 

INFO | jvm 1 | YYYY/MM/DD HH:MM:SS | Dumping heap to ../../logs\java_pid5616.hprof ... 

ERROR | wrapper | YYYY/MM/DD HH:MM:SS | JVM appears hung: Timed out waiting for signal from JVM. 

ERROR | wrapper | YYYY/MM/DD HH:MM:SS | JVM did not exit on request, terminated 

 

STATUS | wrapper | 2017/03/22 15:01:14 | Launching a JVM... 

 

ERROR [localhost-startStop-1] [BrokerService] Temporary Store limit is 51200 mb, whilst the temporary data directory: E:\CA\Catalog\logs\jms-data\brkr-585AE64695A949058FB9DBA1F071EC10\tmp_storage only has 45907 mb of usable space - resetting to maximum available 45907 mb. 

ERROR [localhost-startStop-1] [BrokerService] Failed to start Apache ActiveMQ ([brkr-585AE64695A949058FB9DBA1F071EC10, null], java.io.IOException: Detected missing/corrupt journal files. 4792 messages affected.) 

ERROR [localhost-startStop-1] [ActiveMqBroker] Exception while trying to start JMS broker

Environment:
Service Catalog 14.1 / Service Desk Manager 14.1
Cause:

This seems related to Catalog performance and Catalog accepting the SDM connection rather than a problem with the SDM process itself.

This problem could be caused by a build up of files in the ..\Service Catalog\logs\jms-data folder.

This can lead up to memory build up if there are lot of unconsumed JMS events.

Resolution:

Please take the following steps to remove the files: 

 

1. Stop the CA Service Catalog service

2. Delete Jms-data by making a backup of the folder "..\Service Catalog\logs\jms-data" to a separate folder outside "..\Service Catalog\logs", then delete all the files under "..\Service Catalog\logs\jms-data"

3. Change all from maxBytesLocalHeap="100M" to maxBytesLocalHeap="25M" in ehcache.xml file location \CA\Service Catalog\view\conf of all Node (if all nodes are having the problem otherwise change in problematic node)

4. Start the CA Service Catalog service.

 

Note: Because of the rebuild of the jms message queue, it may take longer for Service Catalog to completely startup. 

 

Once Service Catalog is restarted, monitor the java.exe process for it's CPU and memory usage and check the pdm_catalog_sync daemon process to ensure it is in 'Running' status.

Additional Information:

It might also be needed to increase the Maximum Java Heap Size at the Catalog server to something greater than the default value (check at the viewService.conf file for the parameter 'wrapper.java.maxmemory' currently set by default to '1024' or '1Gb', you might need to increase it to something bigger such as '3Gb' or '4Gb').

Note:

It is possible that in order to accommodate the new memory values, there should also be needed to increase the RAM memory at the servers.