How to purge or reduce the size of a Postgresql APM database and optimize CEM data retention.

    This document provides some guidelines on how to purge and/or reduce the size of an APM database maintained in Postgresql and includes suggestions for optimizing CEM data retention settings.

How to purge or reduce the size of a Postgresql APM database

You can manually execute a "vacuum" process to clean all tables using pgAdmin.

• Open pgAdmin
• Right click on the APM Schema / Database
• Select Maintenance
  
  
• Select Vacuum
• Check to select the "FULL" option
  
  
• Vacuum is an automatic process; however, it runs on tables only when certain threshold values are met.
• Default threshold settings can be changed manually in C:\Program Files\CA APM\database\data or by using pgAdmin:
• In pgAdmin:
  
  
  • Select Tools
  • Select Server Configuration
  • Select postgresql.conf
    
    
• Re-indexing tables may also help reduce the size of the APM database
• In pgAdmin:
  
  
  • Right click on the APM Schema / Database
  • Select Maintenance
  • Select REINDEX
    
    

Suggestions for Optimizing CEM Data Retention Settings

• An APM database can grow rapidly if it contains a large number of invalid defects.
• For testing purposes, try changing the defect retention settings (Incident Management Settings) in the CEM Console, from the default of 7 days to 2 days.
• As aggregation cycles occur overnight, database size will decrease.
  
  
• In the CEM console, ensure that the "Capture Comprehensive Defect Details" check box is not selected on the Setup > Domain page.
• When "Capture Comprehensive Defect Details" is checked, the CEM > Incident Management > Defect details page can display additional data about query and post parameters and request and response bodies.
• For testing purpose, disable it to reduce the amount of data retained.
  
  
• Gradually reduce most Data Retention Settings in the CEM UI > Setup > Domain tab by about 20-30% per day until a satisfactory level is reached.
• "Make Users Inactive after" is the exception. Keeping this setting at a higher value will prevent new user records from being created unnecessarily.
  
  

1. From CA APM Sizing and Performance Guide: APM Sizing and Performance Guide > APM Database CA CEM Data Retention Considerations

APM Database CA CEM Data Retention Considerations
You can configure the length of time that various kinds of monitored user data are retained in the APM database. These settings are ultimately a business decision you base on the requirements at your organization. However, make retention setting decisions with an understanding of the capacity consequences.

Increasing data retention increases the duration of daily statistics aggregation runs. This situation can result in the following performance issues:

• More contention for resources with the Collector that runs the daily stats aggregation service.
• Reduced Collector capacity.

Retention settings have a direct impact on APM database disk space requirements. Periodically monitor available space on the database data disk. If the percentage of available space drops below 25 percent, possible remedial actions include:

• If the APM database logs are not on separate disks, move the logs to separate disks.
• If you plan to have the logs continue to share the data disk, delete old log files.
• Reduce retention periods
• Upgrade to a larger capacity disk subsystem.

One common cause for a database bottleneck is that the resources available to the APM database, primarily memory, are inadequate for the data retention settings.

Reducing the data retention period has temporary side effects on database maintenance tasks. The side effects include longer execution times and higher memory requirements for aggregation and cleanup tasks during the 24-hour period following a reduction in retention times.

Wait from 24 to 72 hours before assessing the effect of a change on retention settings, so that all aggregation, cleanup, and maintenance tasks have run. If older data is being kept for historical purposes and not being included in ongoing analysis or reporting, consider backing up and archiving data. Coordinate these tasks with reduced retention periods.

2. Useful queries to help you troubleshoot the issue.

Run the below query in the Postgres instance to find out the size of the APM database (default name is cemdb):

SELECT pg_database.datname,pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database where pg_database.datname='cemdb';

Run this query under database cemdb to find the largest tables:

SELECT relname,relfilenode,relpages,(relpages*8/(1024*1024)) as disk_space_in_GB FROM pg_class ORDER BY relpages DESC;

Validate whether the tables were auto vacuumed/analyzed or not by running this query under database cemdb:

SELECT relname, n_tup_ins, n_tup_upd, n_tup_del, n_tup_hot_upd, n_live_tup, n_dead_tup, last_vacuum, last_autovacuum, last_analyze, last_autoanalyze FROM pg_stat_user_tables;