This procedure will explain how to reinitialize failed replication in a multi-node cluster during a maintenance window.
The CA API Gateway uses MySQL replication to provide database failover and availability should one Gateway appliance or database server become unavailable or degraded. MySQL replication will ensure that a duplicated copy of a database object is maintained in one or more locations. The Gateway uses a master-master implementation in a multi-node environment to ensure that database changes to one host are replicated to the other database host.
MySQL replication is capable of repairing itself if the other database node is unavailable. Replication can repair itself after a node in the cluster experiences a graceful shutdown, MySQL server shutdown, or a network outage. Replication will fail if a data consistency error is experienced. This article will prescribe how to reinitialize replication in a multi-node cluster.
The following data may be visible when running the SHOW SLAVE STATUS query against the local MySQL database:
The following log entries may be present in the Gateway log files:
2380 WARNING "Error accessing host/database
2381 WARNING "Replication failing for host/database
Note: This process is recommended to be performed during a maintenance window where the Gateway cluster is completely unavailable
This process will result in a significant period of downtime if a large quantity of audits (typically over 1,000,000) are present in the local Gateway database. These audit records can be removed in advance of this procedure via the following articles:
- TEC0000001008 Deleting old audit events from the local Gateway database
- TEC0000001123 Removing audit records from the Gateway database in a multi-node cluster without downtime )
Please contact CA Support if there are any concerns about the length and breadth of the maintenance window
The following procedure should be executed in order to reinitialize replication in a multi-node cluster:
- Back up the database on the PRIMARY node: mysqldump --all-databases > ~/all_databases_`date '+%Y%m%d_%T'`.sql
- Make sure that the last line of the newly created SQL file is "-- Dump completed <date-time>".
- Store the file for future use.
- Stop slave replication on both nodes: mysqladmin stop-slave
- Reset the master configuration on both nodes: mysql -e "reset master"
- Reset the slave configuration on both nodes: mysql -e "reset slave; reset slave all"
- Stop the Gateway service on both nodes: service ssg stop
- Execute the create_slave.sh script on the secondary node: /opt/SecureSpan/Appliance/bin/create_slave.sh
- Provide the fully qualified domain name (FQDN) of the primary node
- Drop the local database copy the remote database
- Execute the create_slave.sh script on the primary node: /opt/SecureSpan/Appliance/bin/create_slave.sh
- Provide the FQDN of the secondary node when prompted
- Decline the request to drop the local database
- Start the Gateway service on both nodes: service ssg start
- Query the status of replication on both nodes: mysql -e "show slave status\G"
NOTE: The "\G" at the end of the line ensures the output is human-readable.
- Verify both nodes return the following lines:
Replication should be re-initialized at this point. The above output indicates that the master/slave relationship is functioning.