The Enterprise Manager uses a "message queue" to communicate back and forth between EM's. So when you encounter a "Timed out adding to outgoing message queue. Limit of XXXX reached" message, this means that the message queue for the EM's communication protocol has filled up and no new messages will be added. This causes the Collector to cease sending out messages and all queries from the MOM to that Collector will error out. Outgoing delivery threads are fixed in number and shared by all the clients. These threads deliver messages from the outgoing message queue to the client, and only when there are no outgoing messages left in current queue move on to the next queue.
Some mail threads may be executing a large number of queries marking them busy on the connected Collectors. In this situation, the mail threads must wait for locks that are required to execute the requests. Since this is also blocked for a response until it times out, all of the mail threads will slow down as a result.
To alleviate this situation, there are 3 properties that work together.
This property is a hidden property inside the IntroscopeEnterpriseManger.properties file. This sets the size of the messaging queue. This property is based on the number of messages, not the size of the message content. If you increase this property value, you should also increase the heap size as the higher the number, the more RAM it uses. Basically the higher the value you use, the more resources it takes up.
This property determines the minimum number of available outgoing delivery threads. We recommend to set the minimum and maximum the same. By doing this, we eliminate the adding/removing of pools when needed. These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottleneck for performance. However, as in the outgoing setting above, the higher the value, the more resources are taken up.
This property determines the maximum number of available outgoing delivery threads. By increasing the maximum pool size, this allows more worker threads to send outgoing messages concurrently. These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottle neck for performance. However, as in the outgoing setting above, the higher the value, the more resources taken up.
The transport.override.isengard.high.concurrency.pool.max.size property in the IntroscopeEnterpriseManager properties file determines the maximum number of available outgoing delivery threads.
In a healthy cluster, the delivery threads run smoothly and it doesn't take long for a filled outgoing message queue to have space freed up for more messages. Delivery threads take messages off the outgoing message queues. If there are fewer delivery threads than active client connections, outgoing messages must wait in the queue, and the queue can reach its capacity. When there is no free space in the outgoing message queue, the Enterprise Manager cannot send requested data back to the client. This can result in serious Enterprise Manager performance problems. Increasing the queue size and thread pool size provides more capacity and can resolve this situation. These important adjustments are required when, sending messages, usually between collectors and MOM, becomes a bottleneck for performance, but care must be taken when making the adjustments.