We are using the Websphere MQ Agent v10.5.2.52 and we noticed that we got java.lang.OutOfMemoryError exceptions.
This was relating to specific configuration in the environment, which created large "com.ibm.mq.jmqi.system.JmqiComponentTls" objects in memory which in turn were holding onto huge "security policy" objects.
It is not a leak and specific to the configuration in the environment.
The heap of the MQMonitor process should be increased, to a minimum 1.5GB.
This is configured in startMQMonitor.sh
Look for the section where minimum and maximum heap are configured
and increase the maximum
Make sure you are using a 64-bit JRE if you want to allocate larger amounts of heap.