Enterprise Manager (EM) Collector is not starting after an upgrade.

Document ID : KB000008739
Last Modified Date : 14/02/2018
Show Technical Document Details
Issue:

  The Collector was upgraded from APM release 10.1 to APM release 10.5.2. After the upgrade, the Collector is not starting. In the IntroscopeEnterpriseManager.log, seeing the following Error message.

 [ERROR] [main] [Manager.com.timestock.tess.util.DbConfigurationUtil] Could not save the Database Configuration.

java.lang.IllegalStateException: com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at com.timestock.tess.util.DbConfigurationUtil.getApmEncryptor(DbConfigurationUtil.java:357)
 at com.timestock.tess.util.DbConfigurationUtil.saveDbConfigurationToXml(DbConfigurationUtil.java:250)
 at com.timestock.tess.web.listeners.CEMWebAppContextListener.contextInitialized(CEMWebAppContextListener.java:72)
 at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1272)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startEnduserWebApps(WebServerEntity.java:782)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startWebServer(WebServerEntity.java:448)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.init(WebServerEntity.java:214)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.<init>(WebServerEntity.java:162)
 at com.wily.introscope.server.enterprise.EnterpriseServer.initialize(EnterpriseServer.java:734)
 at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:382)
 at com.wily.util.ALifeCycle.start(ALifeCycle.java:86)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:325)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:304)
 at com.wily.introscope.server.enterprise.EnterpriseServer.start(EnterpriseServer.java:1598)
 at com.wily.introscope.em.internal.Activator.startEM(Activator.java:119)
 at com.wily.introscope.em.internal.Application.start(Application.java:27)
 at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
 at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
 at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
 at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
 at org.eclipse.core.launcher.Main.main(Main.java:30)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.zerog.lax.LAX.launch(Unknown Source)
 at com.zerog.lax.LAX.main(Unknown Source)
Caused by: com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at com.ca.apm.crypto.util.JCEKSKeystore.loadCrypts(JCEKSKeystore.java:143)
 at com.ca.apm.crypto.jasypt.APMStringEncryptor.<init>(APMStringEncryptor.java:74)
 at com.timestock.tess.util.DbConfigurationUtil.getApmEncryptor(DbConfigurationUtil.java:353)
 ... 40 more
9/24/17 03:23:21.180 PM CDT [ERROR] [main] [org.springframework.web.context.ContextLoader] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.ca.wily.jasypt.APMEncryptablePropertyPlaceholderConfigurer#0' defined in ServletContext resource [/WEB-INF/tess-data.xml]: Cannot resolve reference to bean 'hibernateStringEncryptor' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateStringEncryptor' defined in ServletContext resource [/WEB-INF/jasyptEncryptor.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.ca.apm.crypto.jasypt.APMStringEncryptor]: Constructor threw exception; nested exception is com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:334)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
 at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:651)
 at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1077)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:981)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:487)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
 at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:662)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:461)
 at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
 at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
 at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
 at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:548)
 at org.mortbay.jetty.servlet.Context.startContext(Context.java:136)
 at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1272)
 at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
 at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
 at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startEnduserWebApps(WebServerEntity.java:782)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.startWebServer(WebServerEntity.java:448)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.init(WebServerEntity.java:214)
 at com.wily.introscope.server.enterprise.entity.webserver.WebServerEntity.<init>(WebServerEntity.java:162)
 at com.wily.introscope.server.enterprise.EnterpriseServer.initialize(EnterpriseServer.java:734)
 at com.wily.introscope.server.enterprise.EnterpriseServer.doStart(EnterpriseServer.java:382)
 at com.wily.util.ALifeCycle.start(ALifeCycle.java:86)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:325)
 at com.wily.introscope.server.enterprise.EnterpriseServer.<init>(EnterpriseServer.java:304)
 at com.wily.introscope.server.enterprise.EnterpriseServer.start(EnterpriseServer.java:1598)
 at com.wily.introscope.em.internal.Activator.startEM(Activator.java:119)
 at com.wily.introscope.em.internal.Application.start(Application.java:27)
 at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
 at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:386)
 at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:549)
 at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
 at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
 at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
 at org.eclipse.core.launcher.Main.main(Main.java:30)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.zerog.lax.LAX.launch(Unknown Source)
 at com.zerog.lax.LAX.main(Unknown Source)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hibernateStringEncryptor' defined in ServletContext resource [/WEB-INF/jasyptEncryptor.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.ca.apm.crypto.jasypt.APMStringEncryptor]: Constructor threw exception; nested exception is com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1039)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:985)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:487)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
 ... 54 more
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.ca.apm.crypto.jasypt.APMStringEncryptor]: Constructor threw exception; nested exception is com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:87)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1032)
 ... 62 more
Caused by: com.ca.apm.crypto.InvalidConfigurationException: Key for algorithm AES128 does not exist in keystore!
 at com.ca.apm.crypto.util.JCEKSKeystore.loadCrypts(JCEKSKeystore.java:143)
 at com.ca.apm.crypto.jasypt.APMStringEncryptor.<init>(APMStringEncryptor.java:74)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:148)
 ... 64 more

Environment:
APM Collector is upgraded from APM release 10.1 to APM release 10.5.2 on Linux.
Cause:

  The root cause found to be within the "<EM-Home>/config/internal/server" directory.


  The file keystore.jceks was corrupted.

Resolution:

1. Make a backup of "<EM-Home>/config/internal/server" directory on the problem Collector
2. Copy "<EM-Home>/config/internal/server" directory from a working EM (MOM or Collector)
3. Delete file "<EM-Home>/config/internal/server/tess.internal.properties" to ensure that the Collector gets new unique entity.guid value.
4. Start the Collector

If there is no other EM (MOM or Collector) for same release in your environment, then we suggest installing a EM collector on a machine and make sure it runs properly.
Then copy "<EM-Home>/config/internal/server" directory to the problem EM Collector.

Additional Information:

https://support.ca.com/us/knowledge-base-articles.TEC1786855.html -All collectors are visible in Workstation Investigator & WebView  but some are missing under CEM Setup/Services/Service Configuration & EM Configuration.

https://support.ca.com/us/knowledge-base-articles.tec1248765.html - Seeing this error in multiple collector logs: Unable to create partition for