How to investigate which object made the Data Aggregator's Threshold Limit exceeding

Document ID : KB000010555
Last Modified Date : 14/02/2018
Show Technical Document Details
Introduction:

This document shows you method to find monitoring items which made Threshold Limit exceeded.

And that supplements the content of the following manual. 

Threshold Monitoring and Threshold Limiter Behavior

 

Background:

When the Threshold Limiter is violated, the following error is output to the Data Aggregator karaf log. 

[Example]

WARN | sitory-thread-67 | 2017-07-04 01:30:51,890 | onitoringProcessLimitManagerImpl | onitoringProcessLimitManagerImpl 98 | .ca.im.aggregator.loader | | Threshold Monitoring processing took too long. The system will shut that feature down in 15 minutes if the threshold monitoring continues to exceed capcacity 

INFO | sitory-thread-74 | 2017-07-04 01:30:51,896 | dMonitoringSystemLogNotifierImpl | dMonitoringSystemLogNotifierImpl 48 | .ca.im.aggregator.loader | | Event 2.StateDegraded was generated 

INFO | atcher-thread-63 | 2017-07-04 01:31:05,558 | ThresholdEventItemCache | m.status.ThresholdEventItemCache 91 | .ca.im.aggregator.loader | | violation watcher task completed , current itemsAboveThreshold value={997=pollRate=60000 

lastViolationTime=-1 

violationCount=0, 279741=pollRate=60000 

lastViolationTime=-1 

violationCount=0, 279742=pollRate=300000 

lastViolationTime=-1 

violationCount=0, 1754=pollRate=300000 

lastViolationTime=-1 

violationCount=0} 

 

 

Environment:
RedHat Linux 6.x
Instructions:

By find out the result of this instructions, some customer succeeded to decrease the occurrence of the Threshold Limit excess extremely by changing particular device's Interface polling cycle from one minute to five.  

 

(1) Basic: Show the Polling cycle and the related Metric Family

Execute following command from Data Aggregator's terminal by root account.

/opt/vertica/bin/vsql -U dauser -w dapass -c "select * from item;" -A -F ',' | grep <itemid> 

Note:

* Please use your Data Repository username (-U parameter) and password (-w parameter). 

* The <itemid> is a number  in ahead of "=pollRate=" in the karaf log message. 

 

[Example]

/opt/vertica/bin/vsql -U dauser -w dapass -c "select * from item;" -A -F ',' | grep 997 

997,224,1,NormalizedPortInfo 300000,Represents the event process for metric family {http://im.ca.com/normalizer}NormalizedPortInfo,,1997,227,1,,,, 

 

(2) Advanced: Show the Interface Item name

Execute following command from Data Aggregator's terminal by root account.

/opt/vertica/bin/vsql -U dauser -w dapass -c "select i.item_id, item_name from item_facet f, item i where i.item_id = f.item_facet_id and f.item_id = <itemid>;" -A -F ',' 

Note:

* This SQL refer dual tables by following mixture conditions.

  - Searches with specified item_id from the 'item_facet' table. 

  - Shows the item name from the 'item' table corresponding above result.  

 

[Example]

/opt/vertica/bin/vsql -U dauser -w dapass -c "select i.item_id, item_name from item_facet f, item i where i.item_id = f.item_facet_id and f.item_id = 997;" -A -F ',' 

item_id,item_name 

1970,TenGigE0/6/0/19 

1971,TenGigE0/6/0/20 

1972,TenGigE0/6/0/21 

1973,TenGigE0/6/0/22 

1974,TenGigE0/6/0/23 

(5 rows) 

Additional Information:

[Example of item_facet table retrival]

/opt/vertica/bin/vsql -U dauser -w dapass -c "select * from item_facet where item_id=997;" -A -F ',' 

item_facet_id,item_id,facet_id 

1970,997,1 

1971,997,2 

1972,997,47 

1973,997,93 

1974,997,107 

(5 rows)