JBoss physical memory size is huge , because dat files under adminui_install/server/default/data/derby/siteminder/taskpersistance/seg0 are increasing.
The dat files are database and used to store tasks when user navigates in WAMUI and creates objects.
They are increased each time user push "submit" button to make some tasks in AdminUI.
By "Cleanup Submitted Tasks", dat files size should be reduced.
But, because of product bug, you can't reduced its files size by "Cleanup Submitted Tasks", and it will be fixed in future SP of 12.7 and 12.8.
So, to resolve this issue, you should re-setup AdminUI as below sequence.
1. Stop AdminUI.
2. Delete data folder under adminui_install/server/default.
3. XPSRegClient <superuser name>:<password> -adminui-setup
4. Start AdminUI