.NET Agent OOM, High CPU - Checklist

Document ID : KB000020387
Last Modified Date : 14/02/2018
Show Technical Document Details

First, you need to identity if the issue is related to the instrumentation:

  1. Stop the .NET application.
  2. Open the IntroscopeAgent profile and set introscope.autoprobe.enable=false
  3. Start the process again.

If the problem does not persist, it will confirm that the problem is related to the Agent instrumentation. Enable back introscope.autoprobe.enable=true anddisable the below components:

  1. Disable perfmon collection:
    For v8.0, 9.0: set introscope.agent.perfmon.enable=false in the IntroscopeAgent.profile
    For v9.1: stop the"CA APM Perfmon Collector Service"

  2. Disable any additional custom instrumentation and in addition turn off the following tracing options in the toggles-typical.pbd:

    ServicedComponentTracing
    RemotingClientProxyTracing
    DirectoryServicesTracing
    MessagingTracing
    MessagingTransactionTracing
    SQLAgentDataReaders
    SQLAgentTransactions

  3. If CLR is v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true.

  4. If CEM integration is enabled, try disabling appmap and bizrecording agent properties in introscopeagent.profile.

    Remember these are temporal changes, If the problem does not occur with the base agent, you need to introduce back each component one by one until youreproduce the problem, however If the problem persists, you should contact CA Support and provide the following information

    Enable DEBUG: In the logging.config.xml
    If v9.x, set in the Agent profile : introscope.nativeprofiler.logBytecode=true and introscope.nativeprofiler.logAllMethodsNoticed=true

    Reproduce the issue and collect the below information:

    1. Zipped content of the AGENT_HOME/install/*.logs
    2. Zipped content of the AGENT_HOME/wily/logs folder
    3. AGENT_HOME/wily/IntroscopeAgent.profile
    4. Screenshot of events from Windows Event viewer.
    5. In case of high CPU issue, capture consecutive dumps from a .Net process. If you don't have a preferred tool and setup on their server for such purpose, then below is one suggested approach:

      1. Download procdump.exe from Microsoft. It is a free utility included with Sysinternal or other Windows admin toolkit.
      2. Identify the PID (process ID) of the .NET process, e.g. say 6789 for a w3wp.exe process, that is having unusual high CPU.
      3. Run the following command to capture 3 dumps with 5 seconds apart: e.g.
        Procdump.exe -ma -s 5 -n 3 6789
      4. If the .net app is a 64-bit process, then add -64 to the commandline: e.g.
        Procdump.exe -64 -ma -s 5 -n 3 6789 5. Analyze the dumps.

        NOTE: For any high memory, high CPU, hang problem, you can use Debug Diagnostics Tool from Microsoft to capture user dump, which contains both heap and thread snapshot.
        The following KB has both download link and usage instructions:
        http://support.microsoft.com/kb/2580960