APM Command Center Server stops working - Persistent store is Full

Document ID : KB000109925
Last Modified Date : 07/08/2018
Show Technical Document Details
APM Command Center Server (ACC) server stops working properly.

In the apmccsrv.log the below message is reported:

2018-07-31 14:19:37.773 INFO 18846 [qtp352138147-127854] --- org.apache.activemq.broker.region.Queue : Usage(Main:store:queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E:store) percentUsage=99%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%;Parent:Usage(Main:store) percentUsage=102%, usage=1099791482, limit=1073741824, percentUsageMinDelta=1%: Persistent store is Full, 100% of 1073741824. Stopping producer (ID:DCDEFRDLX006VD-58988-1530018175911-2:10:1:1) to prevent flooding queue://com.ca.apm.acc.cfgsrv.ctrl.377DC56F3E8379A0BD509EFAB98D66750445ED89F5A691D269BE988993E7650E. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 78804s) 
ACC (any version).
The problem is due to the amount of agents (and probably controllers), activeMQ broker runs out of persistent store for messages. It because of a burst of messages at one time (e.g. many agents or controllers connecting) or it may not keep up with arriving number of messages.
1.- Stop ACC server
2.- Adjust the ACC configuration settings in the ACC_HOME/config/apmccsrv.properties by adding the below hidden properties:
a) Increase the size of the persistent store. Default size limit is 1GB, try doubling the size to 2GB.
b) Increase the size of temp store. Default size of 100MB, try setting this to some higher value, e.g. 1GB
If the above doesn’t' help, you can try to clear all the data and let all the agent to reconnect from scratch again
Backup and delete the activemq-data directory in ACC folder.
3.- Start ACC server