AMQ is not starting properly and the logs show nothing Java Runtime:
Oracle Corporation 1.8.0_131 /usr/java/jdk1.8.0_131/jre Heap sizes: current=1005056k free=973598k max=1005056k JVM args: -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote -Djava.io.tmpdir=/opt/ca/axa/apache-activemq-5.11.4/tmp -Dactivemq.classpath=/opt/ca/axa/apache-activemq-5.11.4/conf -Dactivemq.home=/opt/ca/axa/apache-activemq-5.11.4 -Dactivemq.base=/opt/ca/axa/apache-activemq-5.11.4 -Dactivemq.conf=/opt/ca/axa/apache-activemq-5.11.4/conf -Dactivemq.data=/opt/ca/axa/apache-activemq-5.11.4/data Extensions classpath: [/opt/ca/axa/apache-activemq-5.11.4/lib,/opt/ca/axa/apache-activemq-5.11.4/lib/camel,/opt/ca/axa/apache-activemq-5.11.4/lib/optional,/opt/ca/axa/apache-activemq-5.11.4/lib/web,/opt/ca/axa/apache-activemq-5.11.4/lib/extra] ACTIVEMQ_HOME: /opt/ca/axa/apache-activemq-5.11.4 ACTIVEMQ_BASE: /opt/ca/axa/apache-activemq-5.11.4 ACTIVEMQ_CONF: /opt/ca/axa/apache-activemq-5.11.4/conf ACTIVEMQ_DATA: /opt/ca/axa/apache-activemq-5.11.4/data Loading message broker from: xbean:activemq.xml INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@3ffc5af1: startup date [Mon Nov 05 14:00:02 UTC 2018]; root of context hierarchy INFO | PListStore:[/opt/ca/axa/apache-activemq-5.11.4/data/localhost/tmp_storage] started INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/opt/ca/axa/apache-activemq-5.11.4/data/kahadb]
Symptoms of the issue – amq pod shows it was up but the log revealed that amq was not completely on.
Findings - it turned out to be a NFS issue. NFS in general was running properly as other pods are running fine and storing data in NFS folders. We stopped the amq pod deleted the contents of amq nfs folder and then started the pod again. It started creating log file and kahadb but not all the folder contents of kahadb gets created but amq was still stuck. We suspect that when the filesystem was extended few weeks ago, it impacted NFS somehow and after scheduled pod restart this weekend NFS was not able to acknowledge the requests. We did restarted nfs server, rpc-bind, nfs-lock, nfs-idmap, and the pods as well and it did not worked. Finally, we did a system reboot and then amq started working as expected.