How to identify Interface Mapping issues in NFA
In order for NFA to display data for an interface, the interface mapping needs to be correct across all databases in NFA.
NFA 9.2.0 and earlier would sometimes get into a state after NFA detects a router reboot, an IOS update, ifIndex changes, Router Hardware upgrade, or moving a router from one Harvester to another, where the mappings would become out of sync and would stop displaying data for some interfaces.
In NFA 9.2.1, a fix was released to correct most scenarios which can cause this issue, which is why we recommend upgrading to 9.2.1 as soon as possible in your environment.
In short the following ID's all need to match up with each other:
- poller.persistent_map.pstIndex (Harvester Server)
- harvester.interfaces.pstId (Harvester Server)
- reporter.interfaces.ifMapId (Console Server)
- reporter.agents_all_view.persistentIfIndex (Console Server - this is just a view for licensed interfaces, not a table so it will always match the value in reporter.interfaces)
- The ID should also be the same as the folder name in the \NetFlow\Datafiles\ReaperArchive15\<Router IP>\<Interface ID> (Harvester Server in 2 Tier environment and on the DSA in a 3 Tier environment)
Steps to Identify and Resolve:
1) RDP to the NFA Console server and log in to mysql with:
mysql -P3308 reporter
2) The following column's values have to match up with the other databases.
a) If the interface was already Enabled and Licensed and stop showing data, check the agents_all_view view with the query below, using the IP address of your problem device.
Querying the agents_all_view view as opposed otthe reporter.routers table will narrow your focus down to only licensed interfaces and should make it easier to find your problem interfaces in most cases.
Make note of the value for the persistentIfindex value for the problem interface.
select routeraddress, ifindex, persistentIfIndex, name, description, from_unixtime(LastData) from agents_all_view where routeraddress='10.1.1.1';
b) If the interface was never enabled then check the reporter.interfaces table as there will be no entry in the reporter.agents_all_view view.
So you would query the reporter.interfaces table like below using the IP address of your problem router, and again make note of the ifMapId column for your problem interface.
select r.ID as RouterID, i.id InterfaceID, i.ifMapID, i.ifIndex, i.ifdescr, i.enabled, i.updatedon, i.LastFlow from interfaces i join routers r on i.routerId=r.id where r.routeraddress=inet_aton('10.1.1.1');
3) RDP to the Harvester server where you have two databases to check, the Harvester and the Poller.
The following values need to match each other and match persistentIfIndex/ifMapId in the reporter database
4) Query the Harvester database first log in to mysql with:
mysql -P3308 harvester
Then query the harvester database with query below, substituting the IP address of the problem device.
Make note of the pstID for the problem interface which is what needs to match the other databases.
You should be able to identify the interface by the ephId column, which should be the actual ifIndex of the interface.
select * from interfaces where router=inet_aton('10.1.1.1');
5) Then query the poller database in a separate command prompt window.
Log in to MySql with:
mysql -P3308 poller
Then run the query below, substituting the IP address for the problem router.
You should be able to identify the problem interface by the ifindex and the ifdescr columns.
Make note of the routerID column and pstIndex columns.
Also make note if there are ifIndex values of NULL and if any of those NULL ifIndex values have a pstIndex and ifDescr columns matching what the problem interface should have.
select r.address, r.state, p.routerid, p.pstindex, p.ifindex, p.ifdescr from persistent_map p join routers r on r.id=p.routerid where r.address='10.1.1.1';
6) Compare the results of the queries, specifically compare the poller.persistent_map -> harvester.interfaces.pstId -> reporter.agents_all_view.persistentIfIndex or reporter.interfaces.ifMapId
7) Before making any changes please make a backup of the reporter.interfaces, poller.persistent_map, and harvester.interfaces tables tables with the commands below:
On the Console server:
mysql -P3308 -D reporter -t -e "create table backup_interfaces select * from interfaces;"
On the Harvester server:
mysql -P3308 -D poller -t -e "create table backup_persistent_map select * from persistent_map;"
mysql -P3308 -D harvester -t -e "create table backup_interfaces select * from interfaces;"
8) If this was an interface which was already licensed and showing data and suddenly stopped, then you will need to update the harvester and poller databases to match the reporter database.
If this interface was never enabled and licensed proceed to step 9.
a)Update the harvester.interfaces table like below using your problem router IP address. Also where the ephId is the ifindex of the problem interface and the pstId matches the persistentIfIndex from the reporter database found in step 2a.
update interfaces set pstid=35 where inet_ntoa(router)='10.112.4.24'and ephid=12;
b)Before updating the poller.persistent_map table, if you found a NULL value in step 5 for the ifIndex colume which matches pstIndex in step 7a, then you will need to delete that row first.
Use the routerID from step 5 and the pstIndex should be the same as the pstId from step 7a.
delete from persistent_map where routerid=203 and pstIdex=35;
c)Update the poller.persistent_map table like below where the pstIdex matches the pstId from step 7a, the routerID from step 5, and the ifIndex from step 5.
update persistent_map set PstIndex=35 where ifindex=12 and routerid=203;
9) If this interface was never enabled and licensed then you will need to update the reporter.interfaces table to match the harvester.interfaces and the poller.persistent_map tables.
Use the query below where the ifMapId matches the pstId in step 4, the pstIndex in step 5, and the routerID matches the routerId from step 4b. Also the ifIndex should be the actual ifIndex of the problem interface.
update interfaces set ifMapId=35 where routerid=100 and ifIndex=12;
10) Then recycle the CA NFA Harvester and CA NFA Poller service and wait about 15-30 minutes to see if the interface starts showing data again.