Message Queue is showing "too many open files"

Document ID : KB000038983
Last Modified Date : 14/02/2018
Show Technical Document Details

Symptom:

CA Privileged Identity Manager Message Queue is showing this error in its log file


# tail /opt/CA/AccessControlDistServer/ACMQ/acrptmq.log
...
2016-03-10 11:10:24.303 ERROR: Accept() failed: too many open files. Please check per-process and system-wide limits on the number of open files.
...

Hence e.g. on the DMS we see a high number of pending messages or other issues related to malfunctioning Message Queue.


Environment: 

CA Privileged Identity Manager r12.9 SP1, Linux Red Hat
 

Cause:

System wide user and process limits are not honoured by the tibemsd process (which is the main process of the CA Privileged Identity Manager Message Queue).

This can be verified by running these commands in a root shell on the affected box:

# cat /proc/`pidof tibemsd`/limits | grep 'Max open files'
Max open files            1024                 4096                 files

# ls -l /proc/`pidof tibemsd`/fd | wc -l
1025


As it can be seen tibemsd is trying to allocate more file descriptors (here 1025) than its set limits (here 1024)

Workaround:

Please add either one of the two lines to the CA Privileged Identity Manager Message Queue startup script as shown below:

# vi /etc/init.d/ca-acrptmq
...
start() {
...
        # echo -n "Max open files=20000:20000" > /proc/`pidof tibemsd`/limits
        echo -n "Max open files=20000:20000" > /proc/`getmqpid`/limits
        return $RETVAL
}
...


Restart the CA Privileged Identity Manager Message Queue or even reboot the box and verify the error is disappeared and no pending messages are happening.

Additional Information:  

Please see the Linux documentation for how to configure system wide and process specific limits.