bpvirtdb_srvr shutdown and restart repeatedly

Document ID : KB000107072
Last Modified Date : 06/12/2018
Show Technical Document Details
Issue:

After upgrade from 12.6 to 14.x or 17.1, CA Service Desk Manager starts up very slowly. It takes longer time than normal before all processes are connected to domsrvr. Once started up, the bpvirtdb_srvr process is shutdown (it may die due to a crash) and the following error is written to the stdlogs:

bpvirtdb_srvr 10472 ERROR api_msg.c 867 Attempt to get address while not in callback bpvirtdb_srvr 10472 SEVERE_ERROR miscos.c 222 Signal SIGSEGV received - Exiting!

After the shutting down of the process, it is re-started automatically.  Sometimes it is possible to log on through the web interface and to open an existing ticket but more often, once logged on, the Delayed Server Response message is received when you try to perform an action.  If you then look in the stdlog the above error is shown following the bpvirtdb_srvr processes shutting down. 

Note: The issue may occur for a new installation of CA Service Desk Manager, not just an upgrade. 

Environment:

CA Service Desk Manager 14.1.05
CA Service Desk Manager 17.1

Note: In earlier versions, a patch containing the fix to problem USRD 2726 is a prerequisite to following the Resolution.

Cause:

The system has multiple NIC's and the environment variable NX_PICK_FIRST_IPV4 is either not defined, not set to Yes, or is not in effect due to CA SDM not yet recycled after setting it.

To confirm that the issue is related to multiple NIC, you could follow these steps:
 
1. Stop the CA Service Desk Manager service completely, ensuring all CA SDM processes are not running. On Windows, you could stop it from Windows Services and then use Task Manager to double-check that all processes (such as sslump.exe, pdm_d_mgr, etc.) are not listed.
2. Start trace for the file named tcp_port.c by running:  pdm_logstat -f tcp_port.c TRACE
3. Start the CA Service Desk Manager service.
4. Check the stdlog for messages in this format, where the value of <ip_address_or_host_name> is not the same:
<date> <time> <server_name> bpvirtdb_srvr <####> TRACE tcp_port.c <pid> Target ip is <ip_address_or_host_name> and addrInfo->ai_family is 2 
For example:
07/13 11:58:17.16 SDM14PRI bpvirtdb_srvr 5676 TRACE tcp_port.c 564 Target ip is 99.999.999.99, and addrInfo->ai_family is 2 
07/13 11:58:17.17 SDM14PRI bpvirtdb_srvr 5676 TRACE tcp_port.c 564 Target ip is 11.111.111.11, and addrInfo->ai_family is 2 
Note that  the IP addresses in this example are random, just for the purpose of demonstration, and to show merely that they are different in the two messages.
5. To turn off the trace, run: pdm_logstat -f tcp_port.c 


 

Resolution:
To resolve the issue, follow these steps:
  1. ​Take a backup of the NX.env file
  2. Add PICK_FIRST_IPV4 variable in NX.env file using pdm_options_mgr command to set:  @NX_PICK_FIRST_IPV4 = Yes
    • pdm_options_mgr -c -s PICK_FIRST_IPV4 -v Yes -a pdm_option.inst
  3. Take a backup of the NX.env_nt.tpl file.
  4. Add PICK_FIRST_IPV4 variable in NX.env_nt.tpl file using pdm_options_mgr command to set: @NX_PICK_FIRST_IPV4 = Yes
    •  pdm_options_mgr -c -s PICK_FIRST_IPV4 -v Yes -a pdm_option.inst
  5. Recycle CA Service Desk Manager services.

Important Notes:

  • ONLY use this solution after validating that the issue occurs and is due to a multiple NIC environment.


Other Notes:

  • The NX.env file is located in the base CA Service Desk Manager installation directory (referred to as $NX_ROOT, reached by running the command: nxcd).
  • The NX.env_nt.tpl file is located in $NX_ROOT\pdmconf
  • The solution to DE41999 would implement the environment variable as an option in the Option Manager.  This is planned for a future version/patch/roll-up.
Additional Information:
 Service Desk fails to start on a system with multiple NICs      
Implement NX environment variables in AA