Service Catalog stops responding every few hours

Document ID : KB000123202
Last Modified Date : 19/12/2018
Show Technical Document Details
Issue:
Service Catalog Application stops responding  every few hours and the view log is filling up every 20 minutes.
Environment:
service catalog 17.1 
Cause:
A .   the performance tuning on catalog and EEM  are  not done completely
B.    intensive web service calls  made from  service desk  to catalog  because   the sync up settings  from service desk ticket to service catalog request  ( relating to OOTB service desk and service catalog integration ) is turned on ,  it hammers the catalog system   
Resolution:

A.     EEM performance tuning  (  if EEM is hooked up with external LDAP )  : 
1) login EEM UI as EiamAdmin  ( Global )   
2) go to Configure -> User Store ->  Group Configuration    

set Group Resolution Level as "Resolve direct groups"  instead of default "Resolve nested groups"    
set Application Resolution Level as " Resolve direct groups"  instead of default  "Resolve nested groups"

3) go on EEM server to  enable  paging on EEM  :
In server.xml under ...\CA\SC\EmbeddedEntitlementsManager\config\server\ 

change : 
<paged>false</paged> 
to : 
<paged>true</paged> 

save it and  recycle EEM service , and after that , make sure you can login EEM UI as EiamAdmin 

B.   catalog performance tuning on viewService.conf , ehcache.xml , config.properties, server.xml  accordingly 

1)  in config.properties under USM_HOME :
db.max.conn.pool.size needs to set reasonable big . default is 100,  you may change it to 300 or 500 . 

2)  in viewService.conf under USM_HOME\view\conf\ 
set wrapper.java.maxmemory reasonable .  The configurable parameter  defines the max RAM that catalog application can use on the machine .   You need to give enough RAM for catalog application to use .  

3)  in server.xml  under USM_HOME\view\conf\ 
 In http or https connector setting ,  make sure maxThreads  is set reasonably .  For example ,  default is 400 .  You may set it as 800  

4) ehcache.xml  under USM_HOME\view\conf\

Please refer the tecdoc regarding "Form Caching feature in ehcache.xml and its impact":
https://comm.support.ca.com/kb/form-caching-feature-in-ehcachexml-and-its-impact/kb000009946 

Note :  you will need to recycle catalog service after performance tuning above 

C:   Intensive  web service calls  made to  catalog  

In view.log ,  if you find that view.log fills up very quickly with the following error messages repeatedly and frequently   , for example :
-------------------------
... 
2018/12/17 11.49.37.459 ERROR [http-nio-8080-exec-10] [RequestServiceImpl] SOAPREQ010 com.ca.usm.common.CommonException: You do not have access rights to view this request.:276483 at com.ca.usm.billing.ServiceManager.RequestValidator.validate(RequestValidator.java:75) at com.ca.usm.soap.services.RequestServiceImplUtil.validateAccessControl(RequestServiceImplUtil.java:890) 
...
---------------------------

It is likely that other products make web service calls to catalog intensively and constantly ,  to extent , it could hammer the catalog system .   You  need to find out where and which application makes those web service calls to catalog .   A typical example :  if you have OOTB service desk and service catalog  integration  ( i.e.  the integration without using PAM processes )   ,   you may have the attachment, log, note sync up  going on from service desk tickets to catalog requests  on  Service Desk side : 

In Service Desk UI ,   Administration -> Options Manager -> CA Service Catalog , there are  a couple of configurable parameters relating to OOTB  service desk and service catalog integration .    If  those parameters are installed ( turned on ) , one of them is casc_ws_retry : 
============
Enter the no. of retries to synchronize a CA SDM ticket update (activity logs, attachments or status) with CA Service Catalog, in case of failure. Default:0. Enter -1 to try indefinitely. 
Range:-1-50 
============ 

If the  service desk tickets to catalog requests'   sync up  needs to turn on ,   you may reduce the number of retries (casc_ws_retry)  mentioned above  , for example , set as 1 or 2 or 3 ,   but never set it as -1 ( which is going to retry forever ) . 

If your site doesn't really need this sync up at all ,  please simply uninstall all the parameters under  Administration -> Options Manager -> CA Service Catalog via Service Desk UI  and recycle service desk service   to turn off it . 



 

Additional Information:
Service Catalog Performance Tuning :

https://docops.ca.com/ca-service-management/17-1/en/troubleshooting/troubleshooting-ca-service-catalog/ca-service-catalog-performance-tuning

Catalog Form Caching feature in ehcache.xml and its impact:

https://comm.support.ca.com/kb/form-caching-feature-in-ehcachexml-and-its-impact/kb000009946