This is another in a series of knowledge documents to clarify misunderstandings about common APM metrics.
We noticed that Tibco JVM's monitoring heap was not accurate in Introscope10.5.2.24 dashboard under the GC Tab (Bytes Total/Bytes in Use/Heap Used%) when compared to the defined heap size.
APM 10.5.2 Agent
This is working by design.
JVM provides three different metrics:
1. Used Heap: The amount of heap used at the time of polling the metric
2. Max Heap: The maximum amount of heap the JVM can reserves to the process.
3. Committed Heap: The amount of heap that JVM reserves for that process to grow. Based on GC settings, when USED HEAP reaches to a certain percentage of COMMITTED HEAP (say 75%) JVM automatically allocates a higher amount of Committed Heap to grow it further. This can grow until MAX HEAP setting.
When XMS=256, XMX=512, JVM initializes with a COMMITTED HEAP of 256M. USED HEAP is the actual heap occupied which is always less than COMMITTED HEAP. This may be because when process started it may start around 60 MB and keep growing when objects are added to heap. When USED HEAP reach to a certain value (say 200 MB), then JVM increases COMMITTED HEAP to 320 MB from 256 MB.
Now USED HEAP can grow till 320 MB. When again it reaches to certain percentage of RETAINED HEAP, JVM automatically increases RETAINED HEAP further. This continues until 512 MB which is MAX heap setting.
When XMS=512MB, XMX=512MB, then initial COMMITTED HEAP is 512 MB and it cannot grow further but JVM may reduce COMMITTED VALUE to lower value if application is not much using the COMMITTED HEAP.
Incrementing & decrementing logic differs from JVM to JVM (like Oracle Hotspot JVM, IBM JVM etc..) or version to version or GC POLICY to GC POLICY. Even this behavior can be influenced by setting different values to JVM system properties related to heap.
This is how JVM dynamically increases or decreases COMMITTED HEAP. APM shows the COMMITTED HEAP value under Bytes Total metric (not MAX HEAP). When compared to JConsole and the Investigator, one can see that COMMITTED HEAP is matching to Bytes Total metric and USED HEAP is matching to Bytes in Use metric.
There could be small deviations in values when compared to JConsole due different polling intervals at which JConsole & CA Agent polls for those metrics. APM polls for every 15 seconds and JConsole may poll for ever 1 or 2 seconds. We should not understand this as a lag.
Also when comparing KB and BYTES to convert from KB to BYTES it can be achieved by multiplying with 1024 i.e. 509952 KB = 509952*1024 Bytes = 522190848 Bytes.
An Additional Note:
Another important point to note is that we collect those two metrics using JVMs RUNTIME object. JConsole may use MemoryMXBean to collect those metrics. If there is significant discrepancy in these values then it should be JVM bug (highly unlikely it may not be the case).
We also use MemoryMXBean metrics but we show them under GCMonitor node (not GCHeap node).