Third party applications, Windows kernel memory and Spectrum

Document ID : KB000032146
Last Modified Date : 14/02/2018
Show Technical Document Details

Windows operating systems use two finite memory pools for kernel components, based on the amount of physical memory on the machine and the architecture of the Windows version. These are the paged pool, which can be paged out to disk if physical memory is low, and the much smaller non-paged pool (NPP), which guarantees that memory allocated will remain in physical memory. Once non-paged memory has been allocated it cannot be freed. For more information on the Windows kernel address space, see the references below.

What does this have to do with Spectrum? The SpectroSERVER and Archive Manager can be indirectly impacted when non-paged pool memory is exhausted. Third party applications (often antivirus, backup and Host Intrusion Prevention (or HIPS) applications) can consume all available memory in the non-paged pool (NPP). When antivirus and host intrusion applications scan the system, they cache data in the NPP. When most of the non-paged pool memory has been allocated, the Windows kernel (srv.exe process) can't allocate more for itself or for application handles and drivers. This can result in SpectroSERVER and Archive Manager crashes, as well as performance problems in the TCPIP stack.

 

Some examples (not an exhaustive list) of applications that could consume kernel memory:

 

     Veritas
     ARCserve Backup Agent for Open Files or ARCserve Open File Agent
     Inoculan
     McAfee antivirus and/or Host Intrusion Prevention System
     Trend Micro
     Symantec antivirus

 

SYMPTOMS:

 

Remote desktops stop responding. The system is slow overall.

 

The Windows Application event log might include these events:

 

     Event ID: 2022
     Source: Srv
     Type: Error
     Description:
     Server was unable to find a free connection <number> times in the last <number> seconds.

 

     Event ID: 2021
     Source: Srv
     Type: Error
     Description:
     Server was unable to create a work item <number>> times in the last <number> seconds.

 

     Event ID: 2019
     Source: Srv
     Type: Error
     Description:
     The server was unable to allocate from the system nonpaged pool because the pool was empty.

 

It is also possible for third party applications to exhaust the paged pool, in which case you will see this event:

 

     Event ID: 2020
     Source: Srv
     Type: Error
     Description:
     The server was unable to allocate from the system paged pool because the pool was empty.

 

The Spectrum $SPECROOT/SS/VNM.OUT file and Spectrum Control Panel could show this message: "Fatal error unable to allocate heap"

 

The problem is more likely to occur on pre-Vista and 32-bit pre Windows Server 2008 versions of Windows. The problem also might start happening after adding the /3GB switch to the boot.ini. This cuts the available kernel memory in half (see the technet blog).

 

Although there are known non-paged pool leaks in some antivirus and Host Intrusion Prevention software releases, it is the overall allocation of non-paged pool memory that causes the problem. In other words, a leak could be responsible, but your third party software might be working as designed and still exhaust all available NPP memory.

 

REFERENCES:

 

Memory Management - Demystifying /3GB
http://blogs.technet.com/b/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx

 

Memory Management - Understanding Pool Resources
http://blogs.technet.com/b/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx

 

An Overview of Troubleshooting Memory Issues
http://blogs.technet.com/b/askperf/archive/2008/01/25/an-overview-of-troubleshooting-memory-issues.aspx

 

Troubleshooting Server Hangs
http://blogs.technet.com/b/askperf/archive/2007/09/25/troubleshooting-server-hangs-part-one.aspx

 

Memory Management - x86 Virtual Address Space
http://blogs.technet.com/b/askperf/archive/2007/09/28/memory-management-x86-virtual-address-space.aspx

 

Event ID 2019 or 2020 or "Insufficient System Resources" error returned when logging on
Microsoft KB 272568

 

High non-paged pool memory usage on VirusScan Enterprise 8.7i
McAfee KB66905

 

Not enough server storage is available to process this command (when clients attempt to access network shares) note:
McAfee KB 59932

 

Your Windows Server 2003-based or Windows 2000-based computer occasionally becomes unresponsive, and you receive an Event ID 2019 error message in the System log
Microsoft KB 888928

 

Solution:

 

Follow Microsoft KB 177415 to use poolmon to determine which driver is using most of the pool and/or leaking it:
How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks

 

Try disabling the relevant service(s). Sometimes it takes more than disabling services. Microsoft KB 816071 explains how to disable kernel mode filters.

 

Make sure your third-party software is up to date. Contact your system administrator to update the relevant driver(s).

 

Contact the software manufacturer's support.

 

Consider upgrading to 64-bit Windows Server 2008.