Introscope .NET Agent Troubleshooting and Best Practices

Document ID : KB000111638
Last Modified Date : 18/04/2019
Show Technical Document Details
Introduction:
The following is a high-list of techniques and suggestions to employ when troubleshooting the below Introscope .NET common performance and configuration issues:

- NET Agent Installation problems
- Instrumentation not working
- NET app crash, broken or stop responding
- Agent overhead - high CPU and memory consumption
- Application slow response time.
 
Environment:
APM 10.x
Instructions:

1) Make sure you are using the correct .NET Agent installer:

IntroscopeDotNetAgentInstall32_<version>.exe
IntroscopeDotNetAgentInstall64_<version>.exe
 
Use the 64bit Agent installer if you are planning to monitor Windows 64bit OS and you need to monitor both 32 and 64 .NET applications.
 
IMPORTANT: After installation, make sure to restart the .NET application. For example, for IIS run : “iisreset”



2) Review the .NET install log:
 
-If you use the .exe : the installer log is located in the folder from where you launched the installer: IntroscopeDotNetAgentInstall64.log
-If you use the .msi, the installer log is located in the %temp% folder
 
For example
MSI (c) (90:5C) [04:49:31:746]: Product: CA APM .NET Agent (64 bit) -- Installation operation completed successfully.
 
MSI (c) (90:5C) [04:49:31:746]: Windows Installer installed the product. Product Name: CA APM .NET Agent (64 bit). Product Version: 10.5.2.0. Product Language: 1033. Manufacturer: CA Technologies. Installation success or error status: 0.



3) Verify that the correct version of wily.Agent.dll has been registered in the GAC (c:\windows\assembly).
 
For example after installing .NET Agent version 10.5.2
User-added image
If not listed, you need to register it manually, drag and drop AGENT_HOME\wily\bin\wily.Agent.dll into C:\Windows\assembly



4) Verify that the below "environment variables" exist:

Open Command Prompt as Administrator, run: set
The output should be similar to the one below:
 
Cor_Enable_Profiling=0x1
COR_PROFILER={5F048FC6-251C-4684-8CCA-76047B02AC98}
com.wily.introscope.agentprofile=<install_dir>\wily\IntroscopeAgent.profile
 


5) Make sure permissions to the AGENT_HOME have been set accordingly

If you are trying to instrument a Windows service or standalone app, make sure to run:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily <your application>
 
For example:
<AGENT_HOME>\wily\wilypermission.exe <AGENT_HOME>\wily mytestapp.exe



6) Verify that the CA .NET agent has been attached to the monitored .NET process, for example for IIS, w3wp.exe

Run: tasklist /m wily*
Here is the output you should see:
 
Image Name                     PID Modules
========================= ======== ============================================
PerfMonCollectorAgent.exe        1300      wily.NativeProfiler.dll, wily.Agent.dll
w3wp.exe                                    4000      wily.NativeProfiler.dll, wily.Agent.dll,
                                                                  wily.WebServicesAgent.ext.dll
 


7) Make sure the agent has been configured to connect to the right Introscope EM servername and port

Open the IntroscopeAgent.profile, verify the property: agentManager.url.1=localhost:5001



8) If you are using CLR v4, set introscope.nativeprofiler.clrv4.transparency.checks.disabled=true.

The .NET 4 CLR has some additional checks on certain assemblies which may invalidate the instrumented code, thus throwing VerificationException when accessing the application. This agent setting will suppress these checks when set to true.



9) Check that all the Agent logs have been created sucessfully:

After the .NET Agent has been installed, make sure to:
- restart the .NET application. For IIS for example, run : “iisreset”
- make sure to put activity in your .NET application
 
The below logs should be created for each instrumented application:
-AutoProbe.*.log
-IntroscopeAgent.*.log
-nativeprofiler_*.log
 
NOTE: If Native profiler logs are created but no autoprobe & Introscope Agent logs, this means that the agent has not started because it did not find the triggering method
To resolve this issue uncomment "introscope.nativeprofiler.generic.agent.trigger.enabled" and set this property to true to use the generic trigger:

introscope.nativeprofiler.generic.agent.trigger.enabled=true



10) If your application is still not visible from the Workstation or Webview try configuring the agent to monitor all .NET applications

This test will help you confirm if the issue is related to the application.
Open the IntroscopeAgent.profile, comment  the "introscope.agent.dotnet.monitorApplications" property as below:

#introscope.agent.dotnet.monitorApplications=
 

11) You can see the "PerfmonCollectorAgent.exe" but the .NET agent is missing in the Investigator.

Check if there is another .NET profiler installed in the window server(s)

IMPORTANT: Only 1 .NET profiler can run at a time.

The GUID of the  CA APM .NET Agent is {5F048FC6-251C-4684-8CCA-76047B02AC98}


- Open "Command Prompt" as Administrator,
- Run the command:  REG QUERY HKLM /f "COR_PROFILER" /s 

Or redirect the output to a file:  REG QUERY HKLM /f "COR_PROFILER" /s > apm_netagent_regquery_corproofiler.txt

User-added image

In the above example AVICODE, SCOM or Microsoft Monitoring Agent is preventing the .NET Agent to run
SCOM GUID = AD5651A8-B5C8-46ca-A11B-E82AEC2B8E78

To resolve this issue you need to uninstall the "
Microsoft Monitoring Agent" from Add/Remove programs from the affected server(s). 


User-added image


12) Check for possible errors in the the Windows Event viewer > Application log :

a) “Failed to CoCreate profiler” “The profiler was loaded successfully.  Profiler CLSID: '{
D6E0BA92-3BC3-45ff-B9CC-B4B5AB7190BC}
 
the above message indicates that there is another .NET profiler already installed. Only one .NET profiler can run at a time, see previous point.

b) "System.InvalidProgramException: Common Language Runtime detected an invalid program"  "…cannot be activated due to an exception during compilation".

Disable “WCFRuntimeTracing” in the webservices.pbd as below:

#TurnOn: WCFRuntimeTracing


Then restart IIS  or the monitored standalone .NET application.


13) If you are using Ninject, manually update the .NET application configuration to use UseReflectionBasedInjection NinjectSetting.

Make sure the reflection setting is added to all calls to the Kernel init.
 
For example:
public NinjectDependencyResolver()
{
kernel = new StandardKernel(new {UseReflectionBasedInjection = true;});
AddBindings();
}
..
For more information see:
http://stackoverflow.com/questions/5772860/exception-when-using-ninject
http://stackoverflow.com/questions/11989300/running-ninject-3-on-medium-trust-level
http://stackoverflow.com/questions/19770350/invalidprogramexception-profiling-a-webforms-app-using-ninject
 
For exact details how to enable UseReflectionBasedInjection, contact Ninject support or communities: http://www.ninject.org/community.html



14) If the agent is successfully reporting metrics to the Investigator but you don’t see all the expected metrics then check for possible metric clamps:
 
a) From the Workstation or Webview, check if the Agent metric clamp has been reached, expand the branch:
 
 Custom Metric Host (virtual)
   - Custom Metric Process (virtual)
      - Custom Metric Agent (virtual)(collector_host@port)(SuperDomain)
         - Agents
            - Host
               - Process
                   - AgentName
 
looks at the values for : “is Clamped” metric, it should return zero(0)
 
b) Enable Verbose logging and check if the perfom metric clamp has been reached. 

- Open the logging.config.xml, enable verblose logging as below:
<level value="VERBOSE"/>
- Restart IIS or the .NET standalone process .  If you cannot restart the .NET process immediately,  you can force the agent to recognized this change by saving the IntroscopeAgent.profile.

- if the permon metric clamp has been reached you will see in the the Agent log the below message:
 
[VERBOSE] [IntroscopeAgent.PerfMonService] Metric limit of xx has been reached
 
- Uncomment the below property and increase the value as needed, then restart the "CA APM PerfMon Collector Service".

#introscope.agent.perfmon.metric.limit=1000



15) In case of high CPU/Memory usage, slow response time or application no working after the agent has been installed:

You can try to isolate the issue by running the below 5 tests:

TEST#1: Find out if the issue is related to the Agent instrumentation

-Open the IntroscopeAgent.profile , set introscope.autoprobe.enable=false
-Set the environment variable : Cor_Enable_Profiling = 0x0 - this will ensure that the APM agent code is not invoked when .NET CLR is launched
-Stop the Windows Perfmon collector service.
-Restart IIS or the .NET instrumented application.
 
If the problem persists, contact CA Support
If the problem doesn’t’ persists, proceed with TEST#2

 
TEST#2: disable socket and WCF instrumentation:
 
a) enable back instrumentation :
set introscope.autoprobe.enable=true
 
b) Disable Socket instrumentation in the toggles-typical or toggles-full.pbd by commenting the below line:
TurnOn: SocketTracing
 
c) Disable WCF/SOAP header insertion in the webservices.pbd by commenting the below lines:
TurnOn: WCFRuntimeTracing
TurnOn: WebServicesCorrelationTracing
TurnOn: WCFServerFaultTracing
TurnOn: WCFClientFaultTracing
TurnOn: WCFServerTracing
TurnOn: WCFClientTracing
 
NOTE: By disabling WCFRuntimeTracing, the .NET Agent will not insert correlation ID into the WCF/SOAP message headers, if you find this is the root cause of the problem you can try to switch to use HTTP header insertion by uncommenting the below line:
#TurnOn: ClientCorrelationTracing. 


TEST#3: Limit the # of monitored applications, try to monitor only specific application by updating the "introscope.agent.dotnet.monitorApplications" property.

For example, to limit the agent to monitor IIS and the below DummyWinApp.exe application, you would need to update the monitorApplications property as below:
introscope.agent.dotnet.monitorApplications=w3wp.exe,aspnet_wp.exe,DummyWinApp.exe

 User-added image


 
TEST#4: disable SQL instrumentation and the Agent Traces feature:


a) Reduce SQL instrumentation:
 
- reduce the length of the SQL statements. The default maximum length captured by the agent is 999. You can modify this by adding the following line to the IntroscopeAgent.profile file: introscope.agent.sqlagent.sql.maxlength=
 
- disable some sql metrics reporting:
introscope.agent.sqlagent.sql.turnoffmetrics=true
introscope.agent.sqlagent.sql.artonly=true
introscope.agent.sqlagent.sql.turnofftrace=true

b) Disable the entire traces feature temporally by setting (this is hot property)
com.wily.introscope.agent.blame.transaction.doTransactionTrace=false
 

TEST#5: reduce Perform metric collection in IntroscopeAgent.profile to prevent CPU overhead / spikes
 
1.Stop the PerfMonCollectorAgent service
2.Open the IntroscopeAgent.profile, set:

introscope.agent.perfmon.category.browseEnabled=false
Purpose: to confirm whether disabling perfmon category browsing would significantly reduce the CPU overhead

introscope.agent.perfmon.metric.pollIntervalInSeconds=150
Purpose: to confirm if the CPU spikes is related to the polling interval and # of metrics, after increasing the value from 15 to 150 check if the CPU spikes occurs around every 150 seconds

introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,Process|{osprocessname}|*,|ASP.NET*|*
Purpose: you might need to reduce the # of perfmon metrics being gathered. Depending on your application load the default setting could instruct the agent to collect a huge amount of metrics, introscope.agent.perfmon.metric.filterPattern=|Processor|*|*,|.NET Data Provider*|*|*,|.NET CLR*|{osprocessname}|*,|.NET CLR Data|*|*,|Process|{osprocessname}|*,|ASP.NET*|*


3.Start the PerfMonCollectorAgent service and verify if the issue persists.
 
 
What to collect if the problem persists:
 
Enable DEBUG logging in the logging.config.xml and save the IntroscopeAgent.profile so the change in the xml is taken into account.
If the application crashed, set introscope.nativeprofiler.logBytecode=true and introscope.nativeprofiler.logAllMethodsNoticed=true

Try to reproduce the issue and collect the below information

  1. Install logs and Agent logs (if any, AGENT_HOME/wily/logs)

  2. AGENT_HOME/wily/IntroscopeAgent.profile

  3. The result of "systeminfo" command.

  4. The result of "set" command.

  5. Exercise the application, then run "tasklist /m", send the output.

  6. Scrennshot of the C:\windows\assembly folder, listing the wily.*.dll

  7. Screenshot of application events from Windows Event viewer

  8. If the issue is related to the high cup/memory, crash, hung situation, use Debug Diagnostics Tool from Microsoft to capture user dump, which contains both heap and thread snapshots. The following KB has both a download link and usage instructions: http://support.microsoft.com/kb/2580960   
    Follow the steps described in the below link to capture the performance dumps:
    http://blogs.technet.com/b/mspfe/archive/2011/12/01/how_2d00_to_2d00_effectively_2d00_capture_2d00_windows_2d00_memory_2d00_dumps_2d00_pt_2d00_1_2d00_using_2d00_debugdiag.aspx
     
    There are multiple ways to capture dumps on.NET process.  One simple way is to bring up Task Manager, find the .NET process with the memory issue, and then right click on the process to select  Create Dump File  option in its context menu.