The TDM Portal service uses the "Java Service Wrapper" in order to start the Java process on Windows.
The wrapper.conf file contains configuration options for Initial and Maximum memory which can be modified.
The default location of this file is: C:\Program Files\CA\CA Test Data Manager Portal\service\conf\wrapper.conf
The values need to be un-commented and changed to the desired values:
# Initial Java Heap Size (in MB)
# Maximum Java Heap Size (in MB)
The example above, (using the same 16 Gig server) would change the initial value from 256 Megs to 3 Gigs and the Max value from 4 Gigs to 6 Gigs.
Once the file has been modified, save it and recycle the "CA Test Data Manager Portal" service.