Introscope .NET Agent Troubleshooting and Best Practices

Document ID : KB000111638
Last Modified Date : 20/08/2018
Show Technical Document Details
Introduction:
The following is a high-list of techniques and suggestions to employ when troubleshooting the below Introscope .NET common performance and configuration issues:

- NET Agent Installation problems
- Instrumentation not working
- NET app crash, broken or stop responding
- Agent overhead - high CPU and memory consumption
- Application slow response time.
 
Environment:
APM 10.x
Instructions:

1) Make sure you are using the correct .NET agent installer package:

IntroscopeDotNetAgentInstall32_<version>.exe
IntroscopeDotNetAgentInstall64_<version>.exe
 
Use the 64bit Agent installer in case you are planning to monitor x64bit machine and you need to monitor 32 and 64 bit applications.
 
Important: After installation, make sure to restart the .NET application. For IIS run : “iisreset”
 

2) Review the .NET install log:
 
-If you use the .exe : the installer log is located in the folder from where you launched the installer: IntroscopeDotNetAgentInstall64.log
-If you use the .msi, the installer log is located in the %temp% folder
 
For example
MSI (c) (90:5C) [04:49:31:746]: Product: CA APM .NET Agent (64 bit) -- Installation operation completed successfully.
 
MSI (c) (90:5C) [04:49:31:746]: Windows Installer installed the product. Product Name: CA APM .NET Agent (64 bit). Product Version: 10.5.2.0. Product Language: 1033. Manufacturer: CA Technologies. Installation success or error status: 0.
 

3) Verify that the correct version of wily.Agent.dll has been registered in the GAC (c:\windows\assembly).
 
For example when using 10.5.2
User-added image
If not listed, you need to register it manually, drag and drop AGENT_HOME\wily\bin\wily.Agent.dll into C:\Windows\assembly
 

4) Verify that the below environment variables exist:

Open Command Prompt as Administrator, execute the command: set
The output should be similar to the below:
 
Cor_Enable_Profiling=0x1
COR_PROFILER={5F048FC6-251C-4684-8CCA-76047B02AC98}
com.wily.introscope.agentprofile=<install_dir>\wily\IntroscopeAgent.profile
 
NOTE: To disable the .NET agent you can:
a) uninstall the Agent from “Programs and Features”
b) set environment variable : Cor_Enable_Profiling = 0x0 - this will ensure that the APM agent code is not invoked when .NET CLR is launched
 

5) Make sure permissions to the AGENT_HOME have been set accordingly
If you are trying to instrument a Windows service or standalone app, make sure to run:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily <your application>
 
For example:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily mytestapp.exe
 

6) Verify that the .NET agent is attached to the .NET process using : tasklist /m wily*
For example:
 
Image Name                     PID Modules
========================= ======== ============================================
PerfMonCollectorAgent.exe        1300      wily.NativeProfiler.dll, wily.Agent.dll
w3wp.exe                                        4000      wily.NativeProfiler.dll, wily.Agent.dll,
                                                                          wily.WebServicesAgent.ext.dll
 

7) If you are using CLR v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true.
The .NET 4 CLR has some additional checks on certain assemblies which may invalidate the instrumented code, thus throwing VerificationException when accessing the application. This agent setting will suppress these checks when set to true.


8) Check if the Agent logs are being created:

First of all, after the .NET has been installed, make sure to:
- restart the .NET application. For IIS run : “iisreset”
- put activity in your .NET application
 
By default the Agent creates the below standard logs:
-AutoProbe.*.log
-IntroscopeAgent.*.log
-nativeprofiler_*.log
 
NOTE: If Native profiler logs are created but no autoprobe & Introscope Agent logs, this means that the agent has not started because it did not find the triggering method, to resolve this issue uncomment the below property and set it to true to use the generic trigger:
#introscope.nativeprofiler.generic.agent.trigger.enabled=false


9) If instrumentation still does not work, try to configuring the agent to instrument all available .NET applications by commenting the below property, this will help you confirm that the problem is related to the specific .NET application.
 
introscope.agent.dotnet.monitorApplications=
 
Important: Remember that for .NET Standalone applications, you need to include to the above property the application name is available from task manager.
 
For example, if I need to instrument my DummyWinApp.exe, I would need to update the monitorApplications as below
 
introscope.agent.dotnet.monitorApplications=w3wp.exe,aspnet_wp.exe,DummyWinApp.exe
 User-added image

10) Check the the Windows Event viewer for Application log Error messages:

The APM .NET Agent GUID is {5F048FC6-251C-4684-8CCA-76047B02AC98}.
 
Failed to CoCreate profiler” “The profiler was loaded successfully.  Profiler CLSID: '{D6E0BA92-3BC3-45ff-B9CC-B4B5AB7190BC}
 
That indicates that there is another CLR profiler preventing the .NET Agent from probing the .NET process. Once 1 .NET profiler can run at a time, to prevent the conflict uninstall the other .NET Profiler, open the regedit and for “COR_PROFILER” or the CLSID. A common situation is APM conflicting with AVICODE application.  You can try running the below regedit qurery to find out the list of .net profilers
 
-Open Command prompt as Administrator,
-Run: REG QUERY HKLM /f "COR_PROFILER" /s >> apm_netagent_regquery_corproofiler.txt
 
"System.InvalidProgramException: Common Language Runtime detected an invalid program"
"…cannot be activated due to an exception during compilation".


Try to turn off “WCFRuntimeTracing” in the webservices.pbd 
 

11) If you are using Ninject, Manually update the .NET application configuration or implementation to use UseReflectionBasedInjection NinjectSetting. Make sure the reflection setting is added to all calls to the Kernel init.
 
For example:
public NinjectDependencyResolver()
{
kernel = new StandardKernel(new {UseReflectionBasedInjection = true;});
AddBindings();
}

..
For more information see:
http://stackoverflow.com/questions/5772860/exception-when-using-ninject
http://stackoverflow.com/questions/11989300/running-ninject-3-on-medium-trust-level
http://stackoverflow.com/questions/19770350/invalidprogramexception-profiling-a-webforms-app-using-ninject
 
For exact details how to enable UseReflectionBasedInjection, contact Ninject support or communities: http://www.ninject.org/community.html
 

12) If the agent installation is correct, the agent logs are being created, the application is being running successfully but you don’t the expected metrics in the Investigator then the problem could be due to the metric clamp reached
 
a) If you don’t see a specific metric from your application, check the agent clamp from the Metric Browser, expand the branch
 
 Custom Metric Host (virtual)
   - Custom Metric Process (virtual)
      - Custom Metric Agent (virtual)(collector_host@port)(SuperDomain)
         - Agents
            - Host
               - Process
                   - AgentName
 
looks at the values for : “is Clamped” metric, it should return zero(0)
 
b) If you don’t see a specific metrics from your Perfom Collector Agent, check if the problem is related to the perfom metric clamp (introscope.agent.perfmon.metric.limit) default value  is 1000
 
[VERBOSE] [IntroscopeAgent.PerfMonService] Metric limit of xx has been reached
 
You can verify this condition by enable VERBOSE logging in the logging.config.xml, you need to save the IntroscopeAgent.profile for this change in the xml to be consider.
 
 
13) Application is not working due to the with .NET agent overhead 

Try to reduce the Agent instrumentation:

TEST#1:
-disable the entire instrumentation, set introscope.autoprobe.enable=false
-stop the Windows Perfmon collector service.
 
If the problem persists, contact CA Support
If the problem doesn’t’ persists, you need to identify the part of the instrumentation is causing the problem: Agent or Perfmon collector service
 

TEST#2:  Reducing Agent Instrumentation:
 
a) enable back instrumentation : set introscope.autoprobe.enable=true
 
b) try to adjust introscope.agent.dotnet.monitorApplications to monitor only a subset of application
 
c) Turn off Socket instrumentation in the toggles-typical or full.pbd as below :
#TurnOn: SocketTracing
 
d) Disable WCF/SOAP header insertion in the webservices.pbd as below
#TurnOn: WCFRuntimeTracing
#TurnOn: WebServicesCorrelationTracing
#TurnOn: WCFServerFaultTracing
#TurnOn: WCFClientFaultTracing
#TurnOn: WCFServerTracing
#TurnOn: WCFClientTracing
 
By disabling WCFRuntimeTracing, the .NET Agent won’t insert correlation ID into WCF/SOAP message headers, if you identify this is the root cause you can can try to switch to use HTTP header for correlation ID insertion with .NET Agent by enabling the HTTP correlation tracers in webservices.pbd, you need to uncomment the  TurnOn: ClientCorrelationTracing
d) If the you suspect the issue is related to the high # of SQL metrics, try to disable the SQL instrumentation:
 
You can try to:
- reduce the length of SQL statements. The default maximum length captured by the agent is 999. You can modify this by adding the following line to the IntroscopeAgent.profile file: introscope.agent.sqlagent.sql.maxlength=
 
- disable some sql metrics reporting:
introscope.agent.sqlagent.sql.turnoffmetrics=true
introscope.agent.sqlagent.sql.artonly=true
introscope.agent.sqlagent.sql.turnofftrace=true
 
-If the problem persists, you can try turn off SQL tracing in the toggles-typical or full.pbd as below :
 
#TurnOn: SQLAgentCommands
#TurnOn: SQLAgentDataReaders
#TurnOn: SQLAgentTransactions
#TurnOn: SQLAgentConnections
 
e) If you suspect the issue is related to the huge amount of traces being gather, you can try to disable the traces feature temporally by setting
com.wily.introscope.agent.blame.transaction.doTransactionTrace=false
 
This is hot property there is no need to restart the appserver.
 
 
TEST#3: Reduce Perform overhead (CPU overhead/spikes)
 
Verify whether the CPU overhead would be corresponding to certain perfmon reporting/browsing or simply an issue with perfmon counter metric explosion by tuning the perfmon agent configuration in IntroscopeAgen.profile.
 
a) Try setting introscope.agent.perfmon.category.browseEnabled=false in IntroscopeAgent.profile
This will help you confirm whether disabling perfmon category browsing would significantly reduce the CPU overhead
 
b) Try setting introscope.agent.perfmon.metric.pollIntervalInSeconds=150 to see if the CPU spikes would be also occur around every 150 seconds
 
c) By default we gather the below perfom counters, depending on the number of your applications and design this settings could instruct the agent to gather a huge amount of metrics causing a overhead:
 
introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*
 
For testing purpose try:
 
1.Stop the PerfMonCollectorAgent service
2.Open the IntroscopeAgent.profile,
 
introscope.agent.perfmon.category.browseEnabled=false introscope.agent.perfmon.metric.pollIntervalInSeconds=150
introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,Process|{osprocessname}|*,|ASP.NET*|*

3.Start the PerfMonCollectorAgent service
 
 
What to collect if the problem persists:
 
Enable DEBUG logging in the logging.config.xml and save the IntroscopeAgent.profile so the change in the xml is taken into account or triggered inmediately.
 
If the application crashed, enable by code logging, set introscope.nativeprofiler.logBytecode=true and introscope.nativeprofiler.logAllMethodsNoticed=true
 
Try to reproduce the issue and collect the below information

  1. Install logs and Agent logs (if any, AGENT_HOME/wily/logs)

  2. AGENT_HOME/wily/IntroscopeAgent.profile

  3. The result of "systeminfo" command.

  4. The result of "set" command.

  5. Exercise the application, then run "tasklist /m", send the output.

  6. Scrennshot of the C:\windows\assembly folder, listing the wily.*.dll

  7. Screenshot of application events from Windows Event viewer

  8. If the issue is related to the high cup/memory, crash, hung situation, use Debug Diagnostics Tool from Microsoft to capture user dump, which contains both heap and thread snapshots. The following KB has both a download link and usage instructions: http://support.microsoft.com/kb/2580960   
    Follow the steps described in the below link to capture the performance dumps:
    http://blogs.technet.com/b/mspfe/archive/2011/12/01/how_2d00_to_2d00_effectively_2d00_capture_2d00_windows_2d00_memory_2d00_dumps_2d00_pt_2d00_1_2d00_using_2d00_debugdiag.aspx
     
    There are multiple ways to capture dumps on.NET process.  One simple way is to bring up Task Manager, find the .NET process with the memory issue, and then right click on the process to select  Create Dump File  option in its context menu.