Data Aggregator SNMPv3 discovery requires unique snmpEngineId

Document ID : KB000032071
Last Modified Date : 14/02/2018
Show Technical Document Details

PROBLEM:

Discovery of SNMPv3 devices is failing to discover all devices. Only one out of 2 or more of the devices is discovered. When this problem is observed, no clear or obvious error messages are being observed.

For example, here we see Data Collector (DC) debug for one working device, and one that failed:

On-Demand Log for 1.1.1.1 where discovery failed:

Apr 28 00:11:10.528: Sending ICMP request to the ICMP daemon.

Apr 28 00:11:10.530: Sent request to ICMP daemon

Apr 28 00:11:10.566: Received response from ICMP daemon

Apr 28 00:11:10.566: Responded to ICMP Ping.

Apr 28 00:11:10.567: Profile discovery started

Apr 28 00:11:10.567: Starting SNMP profile discovery for profile: [rank=1, id=8206, name=v3]

Apr 28 00:11:19.567: Profile discovery result for profile: [rank=1, id=8206, name=v3]: REQUEST_TIMED_OUT, Response PDU: null

Apr 28 00:11:21.578: Sent request to ICMP daemon

Apr 28 00:11:21.579: Received response from ICMP daemon

Apr 28 00:11:41.590: Sent request to ICMP daemon

Apr 28 00:11:41.590: Received response from ICMP daemon

 

On-Demand Log for 2.2.2.2 where discovery succeeded:

Apr 28 00:11:10.526: Sending ICMP request to the ICMP daemon.

Apr 28 00:11:10.528: Sent request to ICMP daemon

Apr 28 00:11:10.608: Received response from ICMP daemon

Apr 28 00:11:10.608: Responded to ICMP Ping.

Apr 28 00:11:10.609: Profile discovery started

Apr 28 00:11:10.609: Existing profile 8206 will be tried first: true

Apr 28 00:11:10.609: Starting SNMP profile discovery for profile: [rank=5, id=8206, name=v3] (existing profile)

Apr 28 00:11:10.653: Profile discovery result for profile: [rank=5, id=8206, name=v3]: SUCCESS

Apr 28 00:11:12.669: Sent request to ICMP daemon

Apr 28 00:11:12.670: Performing on demand read request = SnmpGetRequest [id=e7396daf-557a-4128-8d36-5c820dab9385, internetAddress=2.2.2.2, hostname=null,

  varbinds=[oid=1.3.6.1.2.1.1.3.0, isList=false, performDelta=false, usesDynamicIndex=false

           oid=1.3.6.1.2.1.1.5.0, isList=false, performDelta=false, usesDynamicIndex=false]

Apr 28 00:11:12.670: Send on-demand request for scalar OIDs Snmp4jRequest [target=org.snmp4j.UserTarget[address=2.2.2.2/161, version=3, timeout=3000, retries=2], pdu=GET[reqestID=31396029, errorStatus=0, errorIndex=50, VBS[1.3.6.1.2.1.1.3.0 = Null; 1.3.6.1.2.1.1.5.0 = Null]]]

Apr 28 00:11:12.711: Received response from ICMP daemon

Apr 28 00:11:12.715: Finished on demand read. Response = SnmpResponse [error=SUCCESS, errorIndex=-1, queriedIP=2.2.2.2]

   SnmpResponseVariable [oid=1.3.6.1.2.1.1.3.0, type=TIME_TICKS, value=46600577, isDelta=false, isList=false, error=SUCCESS, isDynamicIndex=false, indexList=null]

   SnmpResponseVariable [oid=1.3.6.1.2.1.1.5.0, type=OCTET_STRING, value={51,55,53,48,120,45,50,57,54,48,46,110,119,108,97,98,46,98,97,110,107,45,100,110,115,46,99,111,109}, isDelta=false, isList=false, error=SUCCESS, isDynamicIndex=false, indexList=null]

 

Here we see snmpwalk commands for the same IP addresses, run on the CLI of the Data Collector (DC) performing the actual discovery, showing successful responses:

[root@DataCollector ~]# snmpwalk -v3 -u userName 1.1.1.1 system

SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software, s2t54 Software (s2t54-ADVENTERPRISEK9-M), Version 15.1(2)SY4a, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2014 by Cisco Systems, Inc.

Compiled Mon 22-Dec-14 12:24 by prod_rel_team

SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.1.283

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (181920749) 21 days, 1:20:07.49

SNMPv2-MIB::sysContact.0 = STRING: John Doe

SNMPv2-MIB::sysName.0 = STRING: device.domain.com

SNMPv2-MIB::sysLocation.0 = STRING: Building 1

SNMPv2-MIB::sysServices.0 = INTEGER: 78

SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) 0:00:00.00

 

[root@DataCollector ~]# snmpwalk -v3 -u userName 2.2.2.2 system

SNMPv2-MIB::sysDescr.0 = STRING: Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(2)SE7, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2014 by Cisco Systems, Inc.

Compiled Thu 23-Oct-14 12:32 by prod_rel_team

SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.1.516

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (51499524) 5 days, 23:03:15.24

SNMPv2-MIB::sysContact.0 = STRING: John Doe

SNMPv2-MIB::sysName.0 = STRING: device.domain.com

SNMPv2-MIB::sysLocation.0 = STRING: Building 1

SNMPv2-MIB::sysServices.0 = INTEGER: 6

SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) 0:00:00.00

 

So far everything might seem normal. In this scenario, before creating packet captures for analysis, it was determined that the devices at issue are test systems. These systems (three in total), when configured for testing, had the same SNMPv3 configuration copied between systems. This is normally possible for servers where most network devices that support SNMPv3 don't allow for this configuration copying when setting up SNMPv3 access.

 

CAUSE:

The reason for this is because SNMPv3 relies on all devices having a unique SNMP Engine ID value being set on them. This is an industry standard defined in RFC 3411, section 3.1.1.1.

When this is the case, where devices share their SNMP Engine ID, many SNMP based network monitoring tools will discover the first device without problems. Subsequent discovery attempts against devices with the same SNMP Engine ID as the first device discovered will fail because the software recognized the SNMP Engine ID value, recognized the fact that it is already tied to another device, and halts further discovery processing due to the duplicate value observed.

In the end, the CA Performance Center (CAPC), Data Aggregator (DA) and the Data Collector (DC) systems involved will only discover devices with SNMPv3 configurations whose snmpEngineId value does not match a device already discovered that is set with that same SNMP Engine ID value. 

As noted earlier, this is due to an industry requirement defined in RFC 3411, specifically section 3.1.1.1. Direct link to the RFC is:

https://www.ietf.org/rfc/rfc3411.txt

 The specific section we care about here, section "3.1.1.1. snmpEngineID" states:

"Within an administrative domain, an snmpEngineID is the unique and unambiguous identifier of an SNMP engine. Since there is a one-to-one association between SNMP engines and SNMP entities, it also uniquely and unambiguously identifies the SNMP entity within that administrative domain. Note that it is possible for SNMP entities in different administrative domains to have the same value for snmpEngineID. Federation of administrative domains may necessitate assignment of new values."

 

SOLUTION:

To resolve this ensure all devices involved are set with a unique SNMP Engine ID value.

Also note that devices can share SNMPv3 access credentials such as user name, password, authorization type, etc. This allows for the use of fewer SNMP Profiles in CA Performance Manager. While that is the case, the SNMP Engine ID each device has set must be unique per RFC 3411, section 3.1.1.1.