Assisted Triage (AT) produces a large APM Postgres Database

Document ID : KB000010799
Last Modified Date : 14/02/2018
Show Technical Document Details
Introduction:

Assisted Triage functionality was introduced in APM 10.3.

Related tables are:

- at_stories_pivot   -- The ids of all stores (active/inactive)
- at_stories            -- All stories created by assisted triage
- at_evidences       -- All the evidences associated with the story to be analyzed.
 
Environment:
APM 10.5 or lower.
Instructions:

Problem:  For release 10.5 and below there could be a chance of the following tables growing in size significantly based on the customer environment.

Tables:

at_evidences

at_stories

  1.  KB article TEC597519 discusses VACUUMing. This needs to be performed for initial reclaiming of space which is crucial as it can be used while executing DML (data manipulation language). Sometimes, this cannot happen because of a disk space issue. VACUUM needs significant amount of space and often customer raises a concern because he already ran out of space. Therefore, this option may not be doable.
  2. Re-Indexing of these tables before VACUUMing will also help reclaim space and also deleting unnecessary rows occupying space.
  3. The events that comprise the major content of these tables also play a major role in what can be done. (Alerts, UVB, Stalls, Errors etc). Depending on these events, Support can suggest clamps that will take care of any explosion. The below is the query:

select count(*), type from at_evidences group by type; 

 

  1. Also, you need to determine how many day’s worth of data is retained in those tables. By default, APM deletes data that is more than 62 days old. Every day, APM deletes data from at_stories_pivot tables that is older than 62 days and deletes 5 days worth of data from then. That means the offset is 5 days from 62.
  2. The property to do this is : introscope.apm.alert.preserving.time. This is hidden OOB and defaulted to 62. You can reduce this time from 62 to 30 days depending on the client’s interests. But then if the tables do have data more than 30 days old, it does not mean that configuring the value to 30 deletes all the previous data, but it only deletes the offset of data. That means, it deletes data older than 30 to 35 days. Data that is more than 35 days old is still retained in the tables.
  3. To delete any desired data other than what the system can do as explained above, you need to manually delete by writing queries to find out the relevant rows in at_stories and at_evidences tables.
  4. Deleting rows from at_stories_pivot tables will cascade delete from at_stories and at_evidences tables.
  5. Further to this, the below parameters are suggested if alerts and UVB are the primary contents of these tables.

introscope.triage.uvb.gen.minstatustoconsider=3 (The default being 2 caution to 3 Danger)

introscope.triage.alert.gen.minstatustoconsider=3 (Same as above)