Collector failing due to tracer error.

Document ID : KB000030448
Last Modified Date : 14/02/2018
Show Technical Document Details

Symptoms:

When the collector is taking too much time to optimize the Traces Index, there may be this warning in the Collector log: 

[ERROR] [TT Optimization] [Manager.TransactionTracer] Problem performing routine transaction trace storage managememt: java.io.IOException: background merge hit exception:
_1tfr1:c3460513 _1ts76:c304234 _1tyot:c311717 _1u57k:c312634 _1ucrt:c310926 _1ujzk:c309472 _1us3t:c304681 _1v1m2:c305431 _1vcsv:c306646 _1ve3t:c


If optimizing Index Traces keeps increases, this Collector may shutdown with the the following error:
[ERROR] [Manager] Uncaught Exception in Enterprise Manager: In thread Lucene Merge Thread #0 and the message is org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException:
Insufficient system resources exist to complete the requested service
 

Environment:

Introscope from 9.1x to 9.7x.

Workaround:


OPTION 1:
If the Collector is having problems with the transaction traces index file, then shut down the EM, move the (EM_HOME)/traces/index directory, and restart the EM, After doing this, the index should rebuild upon startup.

OPTION 2: The system is running over the open file limit, then increase the size of the open files. These handles are mostly 
FIFO entries which serve as reference points to actual processes. Therefore, it would not affect the file system. These are created
by the Lucene index that APM uses for its traces.