UMP - No metrics available displayed for many nodes

Document ID : KB000034008
Last Modified Date : 14/02/2018
Show Technical Document Details

USM is not displaying metrics/monitoring values for many systems/devices/robots

If USM is not displaying metrics for one or more nodes, then check the ci metric ID mappings between S_QOS_DATA and cm_configuration_item_metric tables. If the metric IDs for a given metric in S_QOS_DATA do not have a corresponding row in cm_configuration_item_metric, they simply won't display in USM.

So... for those hosts, you can run a query like:

select s.ci_metric_id, cm.ci_metric_id from s_qos_data s left join cm_configuration_item_metric cm on s.ci_metric_id = cm.ci_metric_id

One will be the ci_metric_id from S_QOS_DATA, the other will either be the ci_metric_id from cm_configuration_item_metric (which is good), or a NULL (which means the ci_metric_id in S_QOS_DATA is bad and needs to be NULL'd out itself, then the data_engine should be restarted).

The ci_metric_id from S_QOS_DATA and the one from CM_CONFIGURATION_ITEM MUST BE the same!

IF there is a mismatch then the metric won't show in the query at all, it won't just show with a NULL (because our underlying query will only list those rows where they match exactly.)

So your query should be e.g.:

select s.ci_metric_id, cm.ci_metric_id from S_QOS_DATA s left join cm_configuration_item_metric cm on s.ci_metric_id = cm.ci_metric_id? and s.source = 'source you are looking at';

And in addition to looking for NULLs, if an expected metric simply does NOT show up in the result AT ALL, then it also likely means that the ci_metric_id's do not match each other, and so you should reset them as below.

Keep in mind that you may see a ci_metric_id exists in S_QOS_DATA table but is NOT present/does NOT exist in the CM_CONFIGURATION_ITEM_METRIC table. That could also be the cause of 'No Metrics Available.'

So check the rows/two resultant ci_metric_id columns. Any blanks for ci_metric_id in the second column listed in the query output are bad entries. You can then use those 'bad' ci_metric_id values and run:

???? update s_qos_data set ci_metric_id = NULL

or if youre doing this for a specific device, e.g.,

???? update s_qos_data set ci_metric_id = NULL where source = '<source_ip_address>'

Lastly keep in mind that you should make sure that there are no other conflicts with thye data such as target conflicts where data is no longer being saved for a [particular target, e.g., Fa0 and is now being saved for a different target, e.g., Fa0(FastEthernet0) - this case you should either delete or merge the data using the SLM Merge function, to the current target and delete the source data.

In one case I found that the ci_metric_id in S_QOS_DATA which was MBc236e71f6c295ac3ef1f15825fbcea1 did NOT exist in the CM_CONFIGURATION_ITEM_METRIC table.

In this case,

Delete the niscache on the robot where the QoS is coming from, and then restart that robot,

Then Deactivate discovery_server and the nis_server.

'NULL out' the ci_metric_id in s_qos_data, e.g.,

update s_qos_data set ci_metric_id = NULL where source = '<source_ip_of_QOS_data>'

Restart data_engine
Restart discovery_server
Restart nis_server

The discovery_server and nis_server are responsible for cm_configuration_item_metric.

That should force the ci to be put in cm_configuration_item_metric table.

Check the?cm_configuration_item_metric table to see if it is now present, then check the? interface metrics in the USM/USM Interface Tab view once again to see if it still shows "No Metrics Available".

You can use one source/device/robot as a test, then restart the data_engine. This will take some time but the mappings should be corrected after this - but please try it on a single robot first to be sure.

More complete instructions are detailed below:

You may want to try a FULL reset of discovery first to clean up entries/stale data/origins etc.

You can reset discovery in 7.x as per the available Article in the Knowledge base but you may still have some leftover nodes showing no metrics.

Background:

Sometimes the ci_metric_ids from an initial / previous discovery run end up mismatched due to a subsequent discovery after changes have been implemented in the environment, e.g., robot/device name was changed or probe profile names changed etc., so as a result they have to be reset in the tables

Therefore to fix the issue of "No metrics available"

Drag and drop the niscache cleaner probe to the given Robot(s).

?? niscache_clean_1.1.zip

I've attached it to this Article for your convenience.

Restart the Robot Watcher Service (On Linux issue the command ./niminit stop to stop the Robot and then ./niminit start to start the Robot again, then check to make sure the robot is running.

ps -ef | grep nim

controller, hdb and spooler should show up.

update and set the ci_metric id to null for the given system, for example:

?? update s_qos_data set ci_metric_id = NULL where robot = '<hostname>'

Restart the data_engine.

Restart the discovery_server to pick up the new niscache elements

Check for the metrics table information using a similar query such as the one below:

select * from cm_configuration_item_metric where ci_metric_id in (select ci_metric_id from s_qos_data where source like '%abc.def1%')

select * from cm_computer_system where name like '%abc.def1%'


The backend database should then be newly populated but it may take some time. Note that in some cases with large environments it can take over 8 hours as the process is currently sequential (FIFO).

When discovery is finished, you can actively check the discovery_server log and search for the IP/hostname.

Then check the UMP USM node and the metrics should be retrieved.

Note that the above assumes that all correlation settings in the discovery_server are set to true (enabled), e.g., vm_id and robot_device_id, etc., and that the java min and max memory settings are 2048 and 4096 respectively so the discovery_server can handle the load.

Other/Misc (origin cleanup)

select * from cm_computer_system where origin = '<old_bad_origin>'

update cm_computer_system set origin ='<new_origin>' where origin = '<old_bad_origin>'

select * from s_qos_data where origin = '<old_bad_origin>'

update s_qos_data set origin='<new_origin>' where origin = '<old_bad_origin>'


keywords: No metrics available found USM missing device system window intermittently discovery monitoring devices systems robots views display displays view

Attachments:

File Attachments:
TEC000004874.zip