How does data_engine handle retention?
This article assumes that you have the option "Compress data before delete" enabled, which is the default setting. With this option set, data_engine retention is broken down into three categories: raw, hourly and daily. Infrastructure Manager is only capable of changing raw and hourly settings. Admin Console or Raw Configure must be utilized if you want to update the daily retention settings.
To begin with, we need to define a couple of things regarding the interface. The interface describes the raw and historic options as "delete data older than", however it is better to imagine the options as "summarize data older than". Additionally, the interface describes the second option as historic, but it's better to remember it as "hourly".
With that in mind, if we set raw data to 31 days, and remember it as "summarize raw data older than 31 days", we see that data older than 31 days will be aggregated into hourly averages. The next option will be read as "summarize hourly data older than 180 days". This means that we take the hourly data and roll it up into daily averages.
By default, data_engine tends to be rather aggressive with the amount of data that it tries to retain. Bear in mind that the retention values are additive. With the default values, we store 180 days of raw samples, 720 days of hourly samples and 744 days of daily samples for a total of 1,644 days. With a five minute polling interval, this works out to 69,864 stored samples per monitored metric. It is not uncommon for an installation to have 500,000 metrics or more.
In practice, it may not be necessary for you to keep six months of granular data; you may only be concerned with 30-45 days of highly granular data for SLA purposes, for example. Hourly samples are kept for 720 days. Again, it may not be necessary for you to retain almost two years of hourly data. A more reasonable setting might be six months to a year. Daily averages, on the other hand, are not as much of a concern for storage since it works out to one sample per day. So, 744 days of retention works out to 744 stored samples per monitored metric. If historic data needs to be retained for a lengthy period, then this is the number that should be increased most liberally.
With an environment monitoring 500,000 metrics, tuning the data retention settings in this manner can mean the difference between storing 35,000,000,000 records and 7,000,000,000. This will have a direct impact on reporting and maintenance performance.