Configuring API Gateway to Avoid Auditing/Processing Shutdown, Caused by Audits Exceeding Storage Threshold.

Document ID : KB000008534
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

API Gateway stops auditing and/or processing requests.

  • Observed in /var/log/mysqld.log

170901 21:04:57 [ERROR] /usr/sbin/mysqld: The table '#sql-977_90eda7' is full
170908 21:05:22 [ERROR] /usr/sbin/mysqld: The table '#sql-977_93d076' is full
170909 21:05:29 [ERROR] /usr/sbin/mysqld: The table '#sql-977_943866' is full
170914 21:06:34 [ERROR] /usr/sbin/mysqld: The table '#sql-977_966c1e' is full

  • Observed in /opt/SecureSpan/Gateway/node/default/var/logs/ssg_0_0.log

2017-09-15T16:27:37.132-0400 WARNING 19485495 com.l7tech.server.audit.AuditArchiver: 2205: Audit Archiver error: Receiver not enabled.
2017-09-15T16:27:37.132-0400 WARNING 618 com.l7tech.server.message: Message was not processed: Undefined (-1)
2017-09-15T16:27:37.133-0400 WARNING 618 com.l7tech.server.audit.AuditRecordManagerImpl: Unable to save AuditRecord: Database is full!
2017-09-15T16:27:37.255-0400 WARNING 822 com.l7tech.server.SoapMessageProcessingServlet: Message processing suspended by the Audit Archiver
2017-09-15T16:27:37.257-0400 WARNING 822 com.l7tech.server.MessageProcessor: 4: Message processing suspended: Audit records database disk usage exceeded emergency limit (90 >= 90). Exception caught!

Environment:
- CA API Gateway, all form factors.- Audits stored internally in MySQL Database.
Cause:

API Gateway stops auditing and/or processing requests, due to audits going over the set storage threshold and triggering archiver shutdown policy.

  • Once the amount of space taken by audit records in the database reaches the threshold defined in audit.archiverShutdownThreshold Cluster-wide Property, Gateway will stop auditing and/or processing requests, based on the strategy outlined in the audit.managementStrategy Cluster-wide Property.
  • Default value for the threshold defined in audit.archiverShutdownThreshold Cluster-wide Property is %90 of the smaller value between the size of the logical volume holding the Database file, and the maximum size for the Database file in MySQL settings.
  • Default value for audit.managementStrategy Cluster-wide Property is set to 'STOP', which results in the gateway stop processing requests and terminating audit logging. 
  • By default, database file is stored under /var/lib/mysql/ibdata, on the /dev/mapper/vg00-lv_db logical volume.
  • Maximum size for the Database file in MySQL settings is defined in /etc/my.cnf
    • innodb_data_file_path=ibdata:100M:autoextend:max:15851M
Resolution:

Fix

  • In the case where maximum database size in MySQL settings is smaller that the size of the logical volume hosting the Database file, increasing value for autoextend:max in /etc/my.cnf, and restarting MySQL and SSG services result in the gateway resuming processing of requests and logging auditing.
  • Doing a clean up on the database through removing audit records, as described in  TEC0000001123, and restarting MySQL and SSG services result in the gateway resuming processing of requests and logging auditing.

Prevention

Additional Information:

Using instructions provided in TEC1981945, information obtained could be used to determine how to manage the removal of older audit records:

  • The size of the Databases in the MySQL instance
  • The amount of data per table
  • The newest and oldest audit record
  • The number of audit records it contains
  • The date range of these records