CA API Developer Portal containers are not coming up

Document ID : KB000112262
Last Modified Date : 29/08/2018
Show Technical Document Details
Issue:
API Portal 4.2 does not come up after a power outage and restart.

We are using MYSQL database. 

We see the following error messages in the portal logs :  

| 2018-08-20 15:56:13.854  WARN [tenant-provisioning-service,,,] 9 --- [           main] c.c.a.p.tenant.config.LiquibaseConfig    : Failed to get DB connection on attempt 0 of 5 [An attempt by a client to checkout a Connection has timed out.], retrying in 5s
 | 2018-08-20 15:56:20.318  INFO [tenant-provisioning-service,,,] 9 --- [           main] liquibase                                : Waiting for changelog lock....

So the portal tenant is waiting for changelog lock :
Environment:
API Portal 4.2
With MYSQL database
Cause:

On powerdown and restore the database log lock status can be in set in an unavailable state.  

The portal documentation : 
Has some troubleshooting steps: 
 
mysql> use portal;
mysql> UPDATE DATABASECHANGELOGLOCK SET locked=0, lockgranted=null, lockedby=null WHERE id=1;
 
But there are a number of databases in use, and the locked log setting may not be in the "portal" database.
Resolution:
But full resolution means checking ALL the mysql databases to see if any of them have DATABASECHANGELOGLOCK entries locking access to the database. 

In this particular case we find the tenant provisioning database has an entry in DATABASECHANGELOGLOCK locking access to the database. 

The below steps solved the issue:

mysql>   use TENANT_PROVISIONING ;
mysql> SELECT * FROM DATABASECHANGELOGLOCK; UPDATE DATABASECHANGELOGLOCK SET locked=0, lockgranted=null, lockedby=null WHERE id=1;

But all databases should be checked for LOGLOCK entries.