Team Center - Blank Page after Login, Status code 503, SSLHandshakeException

Document ID : KB000125671
Last Modified Date : 01/02/2019
Show Technical Document Details
Issue:
After applying 10.7 HF#24 or any higher version (including SP3) you noticed that Team Center is not longer working as expected

Configuration:
You are using HTTPS to connect to Webview (introscope.webview.enterprisemanager.rest.base property in IntroscopeWebView.properties)
And using the self-signed certificate installed by the product's EM installer and WebView

Symptoms:
- Blank Page after Login
- No data displayed
- Unknown error
- Notifications:
             Error retrieving lazy upgrade status. Status code: undefined
             Error retrieving permissions. Status code: 503
             Error retrieving settings object(s) of the type title-order. Status code 503


User-added image

- the below exception is reported in the Webview log:

Unable to establish connection with remote resource at https://<em-hostname>:8444/apm/appmap/private/metric/batch! 
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching <em-hostname> found 
Environment:
APM 10.7 HF#24 or any higher version (including SP3)
Cause:
You are using the self-signed certificate installed by the product's installer and is being rejected now as it is invalid.
 
Resolution:
Upload a valid security certificate to the Enterprise Manager keystore, which is located in <EM_HOME>\config\internal\server\keystore. 
By default, this keystore contains a self-signed certificate that is not trustworthy. Upload a certificate that a recognized Certificate Authority (CA) has signed.  


Workaround

Create a self-signed certificate with the correct EM server hostname
Below 2 examples explaining the process, for more information about creating a certificate contact your IT Security Team.

In this example:
- EM hostname  = apmrh6u2b4a.local.int
- By default "<EM_HOME>/config/internal/server/keystore" password is "password"
- By default "<EM_HOME>/jre/lib/security/cacerts" password is "changeit"


Example #1: If EM and Webview are installed in the same directory

In this example, introscope is installed in /introscope

Step 1: Backup keystore original files:

/introscope/config/internal/server/keystore,
/introscope/jre/lib/security/cacerts

Step 2: create self-signed certificate

cd /introscope/config/internal/server
"/introscope/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=apmrh6u2b4a.local.int"
"/introscope/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt 
"/introscope/jre/bin/keytool" -importcert -keystore "/introscope/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit

Step 3: update introscope EM and webview jetty xml files to use the new certificate

By default, Jetty is configured to start a single SSL connector on port 8444. This connector is used for communication with Agents and for HTTP communication over SSL, including Public REST API.
You can reuse the default connector or create a new one, in this example, we are reusing the existing one:

Open /introscope/config/em-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
 <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="TrustStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

Open the /introscope/config/webview-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="TrustStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

NOTE: In case you need to delete the above created jettyssl certificate, you can use the below commands, for more information contact your IT Security team

"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/jre/lib/security/cacerts" -storepass changeit
"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password

Step 4: Start EM and Webview

  
Example #2: If EM and Webview are installed in separate directories

In this example:
- introscope is installed in /introscope
- webview is installed in /introscopeWebview

Step 1: Backup keystore original files:

/introscope/config/internal/server/keystore,
/introscope/jre/lib/security/cacerts
/introscopeWebview/config/internal/server/keystore
/introscopeWebview/jre/lib/security/cacerts

Step 2:  create self-signed certificate

cd /introscope/config/internal/server
"/introscope/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=apmrh6u2b4a.local.int"
"/introscope/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt 
"/introscope/jre/bin/keytool" -importcert -keystore "/introscope/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit

Step 3: update introscope EM jetty xml files to use the new certificate

By default, Jetty is configured to start a single SSL connector on port 8444. This connector is used for communication with Agents and for HTTP communication over SSL, including Public REST API.
You can reuse the default connector or create a new one, in this example, we are reusing the existing one:

Open /introscope/config/em-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
 <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="TrustStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

Step 4: Import certificate into Webview JRE and update the Webview keystore.

"/introscopeWebview/jre/bin/keytool" -importcert -keystore "/introscopeWebview/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit
cp /introscope/config/internal/server/keystore /introscopeWebview/config/internal/server/keystore


Step 5: update webview jetty xml files to use the new certificate
Open the /introscopeWebview/config/webview-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="TrustStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                   ...

NOTE: In case you need to delete the above created jettyssl certificate, you can use the below commands, for more information contact your IT Security team

"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/jre/lib/security/cacerts" -storepass changeit
"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password
"/introscopeWebview/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscopeWebview/jre/lib/security/cacerts" -storepass changeit
"/introscopeWebview/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscopeWebview/config/internal/server/keystore" -storepass password

Step 6: Start EM and Webview



Important Reminders

1. If you previously used  <Set name="verifyHostnames"> please note that this property is NO longer valid, it will be removed from the jetty xml file after the hotfix is installed
If you add the property back it will prevent the EM to load the jetty config file and the below exception will be reported in the EM log during EM startup:

1/29/19 04:20:23.437 PM EST [ERROR] [main] [Manager.EMWebServer] Error loading /introscope/./config/em-jetty-config.xml
org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$1: class org.eclipse.jetty.util.ssl.SslContextFactory.setVerifyHostnames(class java.lang.String)
    at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.set(XmlConfiguration.java:661)

    
2. If during EM startup you notice the below exception in the EM log, you must reenter the credentials in the tess-db-cfg.xml

1/29/19 04:23:21.184 PM EST [ERROR] [main] [org.springframework.web.context.ContextLoader] Context initialization failed
com.ca.apm.crypto.InvalidConfigurationException: Given final block not properly padded


Open the EM-HOME/config/tess-db-cfg.xml
Update "plainTextPasswords" to false  and reenter the "hibernate.connection.password"
Restart the EM 
Additional Information:
https://comm.support.ca.com/kb/_apm-10-7-hotfixes/KB000105898
https://comm.support.ca.com/kb/_em-failed-to-start-with-invalidconfigurationexception-in-apmenterprisemanager-properties/KB000108026