Agile Central - On-premises: The Reporting Service Is Currently Unavailable

Document ID : KB000094782
Last Modified Date : 12/09/2018
Show Technical Document Details
Issue:
The Agile Central on-premise appliance is reporting the following error:
The Reporting Service Is Currently Unavailable
Please check back shortly.
User-added image

 
Cause:
This error can be caused by a custom report that causes the analytics service to run out of memory.  Some poorly written queries can cause the analytics service to use it's memory and not be able to correctly clean up after itself.

This can be verified by gathering a log bundle from the Control Panel and looking at the analytics-startup.log file.
Searching for "Caused by" will typically locate the first instance of the problem and the message may appear as follows:
Caused by: java.lang.OutOfMemoryError: Java heap space
Resolution:
Immediate remediation is to restart the Agile Central services.

The best course of action is to locate the offending custom report and ensure the query is not causing performance issues.

However, if this report can not be rewritten, then it may be necessary to increase the memory assigned to the analytics service.  You are welcome to open a support case to perform this action if you are not comfortable working on the command line or if you do not have root access to your appliance.

Increasing analytics memory allocation
  1. Log in as root to your appliance
  2. Edit /home/appuser/appserver-config-latest/analytics/etc/analytics.cfg using your preferred text editor
    • Example: # nano /home/appuser/appserver-config-latest/analytics/etc/analytics.cfg
  3. Locate the line that reads: _X("mx") { analytics_max_memory || "512m" }
    • There are two lines that are very similar; one is min and the other is max.  Make sure to change the one that says "max"
    • Note that there are multiple memory lines in this file, but you want the first occurrence in the ":base" environment profile:
    • environment(:base, :default=>true, :java_home => "/usr/lib/jvm/jdk1.7.0_80") {
  4. Change 512m to 1024m
  5. Restart the Agile Central services
  6. Verify that the setting has been updated using 'ps'
    1. # ps ax | grep analytics | grep 1024m
    2. 20475 ?        Sl    92:34 /usr/lib/jvm/jdk1.7.0_80/bin/java -Dkill-tag=false -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5101 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -DXSSGUARD_ENABLED=true -Danalytics.jdbc.username=rallyapp_analytics -Danalytics.jdbc.password=rallyapp_analytics -Danalytics.jdbc.service.name=@//localhost:1521/f4wp1 -Danalytics.bootstrap.db=true -DANALYTICS_BIND_ADDRESS=rallyonprem-oraclese -DANALYTICS_SERVER_PORT=8080 -DLOG_DIR=/home/appuser/domains/analytics/logs -Danalytics.log.dir=/home/appuser/domains/analytics/logs -Dlogback.configurationFile=file:///home/appuser/appserver-config-latest/analytics/etc/logback.xml -Djetty.home=/home/appuser/appserver-config-latest/jetty-9 -Djetty.base=/home/appuser/appserver-config-latest/jetty-9 -Drally.jetty.home=/home/appuser/appserver-config-latest/analytics -Djetty.class.path=/home/appuser/appserver-config-latest/migration/ojdbc16.jar -Djava.security.egd=file:///dev/urandom -Djava.io.tmpdir=/home/appuser/appserver-config-latest/analytics/tmp -DSTACK=onprem -Xms1024m -Xmx1024m -Xloggc:/home/appuser/domains/analytics/logs/gc.log -XX:MaxPermSize=256m -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:+UseCompressedOops -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:-TraceClassUnloading -jar /home/appuser/appserver-config-latest/jetty-9/start.jar /home/appuser/appserver-config-latest/analytics/etc/jetty.xml
       
It is not recommended to increase beyond 1024m as the time spent during memory cleanup can begin to negatively impact the performance of the service.

If you continue to experience errors, then the next course of action would be

 
Additional Information:

How to restart Agile Central Services
How to create a log bundle

Keywords:

Rally