Seeing Log Messages "skip old mtp file with file timestamp"

Document ID : KB000016214
Last Modified Date : 14/02/2018
Show Technical Document Details
Introduction:

 Seeing Log Messages revealing the problem:

 

Sat Sep 17 12:18:26 2016 17295   w0: AdapterManager: feedId: 0 skip old mtp file with file timestamp: 1506182354 (seconds), pkt timestamp: 1506182352 (seconds), now: 1506183506(seconds), limit period: 15(minutes), file: /nqtmp/tim/0/w0/1506182352_0.pcap
Sat Sep 17 12:18:26 2016 17295   w0: AdapterManager: feedId: 0 skip old mtp file with file timestamp: 1506182355 (seconds), pkt timestamp: 1506182353 (seconds), now: 1506183506(seconds), limit period: 15(minutes), file: /nqtmp/tim/0/w0/1506182353_0.pcap
Sat Sep 17 12:18:26 2016 17295   w0: AdapterManager: feedId: 0 skip old mtp file with file timestamp: 1506182356 (seconds), pkt timestamp: 1506182354 (seconds), now: 1506183506(seconds), limit period: 15(minutes), file: /nqtmp/tim/0/w0/1506182354_0.pcap
Sat Sep 17 12:18:26 2016 17295   w0: AdapterManager: feedId: 0 skip old mtp file with file timestamp: 1506182357 (seconds), pkt timestamp: 1506182355 (seconds), now: 1506183506(seconds), limit period: 15(minutes), file: /nqtmp/tim/0/w0

 

Question:

  The MTP seems to be running, but from time to time the below log-message shows up and then in some cases, no   more statistics are computed.

 w0: AdapterManager: feedId: 0 skip old mtp file with file timestamp: 1506182354 (seconds), pkt timestamp: 1506182352 (seconds), now: 1506183506(seconds), limit period: 15(minutes), file: /nqtmp/tim/0/w0/1506182352_0.pcap

 How can we resolve it?



 

Environment:
CA APM 10.xMTPTIM
Answer:

   Check the CPU utilization and the incoming traffic. Most likely, all TIM workers are using up to 100% CPU and the tmpfs/tim filesystem is using more than 50% space. As a general rule, when the log message shows up and some or all of the TIM Workers are using 100% CPU, the cause is overload on the CPU side.


  The TIM Workers are overloaded, and are not able to actually process the provided packet captures in time to continue working on their pcap queue. The worst-case scenario is that the tmpfs/tim RAM file system fills up, and the nqcapd or apmpacket are not able to write the files into that temporary space anymore, hence discarding the packets. This packet drop will not be displayed in the protocolstats collected by the TIM.

  Possible remediation is to actually provide more CPU cores to the TIM workers (if there are cores available), or split the traffic to have more TIM workers handling the traffic.

 

Important Note:
Increasing the tmpfs/tim RAM file system is not recommended. It will only delay the moment the adapter manager will start dropping traffic due to overload.

 

Additional Information: