portaldb deployment failed with misleading "invalid memory" error in CA API developer portal 4.2

Document ID : KB000115524
Last Modified Date : 20/09/2018
Show Technical Document Details
Issue:
Log of the portaldb or portaldb-slave container shows error "FATAL: invalid memory request size xxxx", but the docker host still has enough memory.
The portaldb/portaldb-slave are using default postgres database.
Resolution:
The error message could be misleading. In fact, you may see another HINT message in the log,
portal_portaldb-slave.1.xxx    | HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.

For portal 4.2, the memory limit of portaldb containers are set as the same as the RAM of docker host. ie. as soon as docker host still has free memory, the "invalid memory" message is more likely due to the corrupted data. In that case, we may need to restore the database with previous backup. 

For more details about backup/restore, please refer to,
https://docops.ca.com/ca-api-developer-portal-enhanced-experience/4-2/en/set-up-and-maintenance/maintain-the-api-developer-portal/back-up-and-restore-internal-database

Here are some commands may be helpful on trouble shooing,
1. check memeory usage
free -h 
docker stats --no-stream 

2. export portaldb logs
1). docker service logs portal_portaldb-slave > portal_portaldb-slave.log 2>&1 
2). docker service logs portal_portaldb > portal_portaldb.log 2>&1 

 
Additional Information:
Some discussions on postgres,
https://www.postgresql.org/message-id/54889560.6000501%40fuzzy.cz
https://www.postgresql.org/message-id/20160212074625.a8255b924c07c3f716dc191a%40potentialtech.com