APM Command Center Server stops working - Persistent store is Full

Document ID : KB000109925
Last Modified Date : 07/08/2018
Show Technical Document Details
Issue:
APM Command Center Server (ACC) server stops working properly.

In the apmccsrv.log the below message is reported:

2018-07-31 14:19:37.773 INFO 18846 [qtp352138147-127854] --- org.apache.activemq.broker.region.Queue : Usage(Main:store:queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E:store) percentUsage=99%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%;Parent:Usage(Main:store) percentUsage=102%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%: Persistent store is Full, 100% of 1073741824. Stopping producer (ID:DCDEFRDLX006VD-58988-1530018175911-2:10:1:1) to prevent flooding queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 78804s) 
Environment:
ACC (any version).
Cause:
The problem is due to the amount of agents (and probably controllers), activeMQ broker runs out of persistent store for messages. It because of a burst of messages at one time (e.g. many agents or controllers connecting) or it may not keep up with arriving number of messages.
Resolution:
1.- Stop ACC server
2.- Adjust the ACC configuration settings in the ACC_HOME/config/apmccsrv.properties by adding the below hidden properties:
 
a) Increase the size of the persistent store. Default size limit is 1GB, try doubling the size to 2GB.
com.ca.apm.acc.broker.storeUsageLimit=2147483648
 
b) Increase the size of temp store. Default size of 100MB, try setting this to some higher value, e.g. 1GB
com.ca.apm.acc.broker.tempUsageLimit=1073741824
 
If the above doesn’t' help, you can try to clear all the data and let all the agent to reconnect from scratch again
Backup and delete the activemq-data directory in ACC folder.
 
3.- Start ACC server