NH_USER library problem causes SSH failure in network

Document ID : KB000010021
Last Modified Date : 14/02/2018
Show Technical Document Details
Introduction:

After a recent system patching the $NH_USER lost the ability to ssh/scp from one eHealth server to another in the cluster. This issue is caused by the NH_USER profile not using OS provided binaries.

The error seen trying to ssh between servers as NH_USER is:

[NH_USER@eH_HostName ~]$ ssh ServerName 

ld.so.1: ssh: fatal: relocation error: file /usr/bin/ssh: symbol ENGINE_load_pk11: referenced symbol not found 

Killed 

This is the current value for the LD_LIBRARY_PATH variable for NH_USER:

[NH_USER@eH_HostName ~]$ env | grep LD_LIBRARY_PATH 

LD_LIBRARY_PATH=/ehealth/eh63/client11/lib:/usr/dt/lib:/ehealth/eh63/lib:/ehealth/eh63/web/httpd/bin 

Background:

Support View of this situation:

The path /opt/csw indicates use of Blastwave. The reason for using Blastwave in preference over native Solaris tools is more up-to-date patches and more easily managed with "apt equivalent” method of installs, apt being the native Debian packager. Note it appears Blastwave rolled up in 2012 and they have been replaced with OpenCSW. See the OpenCSW web site for more information.

There are additional indications that using the OpenCSW version of SSH which will respect a lot of paths for things such as LD_LIBRARY_PATH as opposed to the native Solaris SSH stack, which doesn't. 

What has been done to resolve this is technically correct to ensure the OpenCSW versions of libraries take priority over others we provide. The assumption here is that our version of Apache will work properly with newer zlib compression libraries provided through OpenCSW. This may technically work but is technically unsupported. It may work but is unsupported due to lack of formal testing to provide support on this use case.

This is the risk with using OpenCSW as a layer over native Solaris. This is similar to use of CentOS instead of RHEL and the related policy around use of non-standard Linux variants. It is technically possible but RHEL is the tested and supported platform.

In this situation it is wonderful that the solution is working for both eHealth and SSH functionality. Though we do have to note that if this change does impact other product areas, if it is proven as the cause for those issues, we will request and require a change back to default.

Environment:
All eHealth installations on Linux
Instructions:

Solution in place that works is as follows:

1. Modified the below in the "/opt/eHealth/nethealthrc.sh" file:

LD_LIBRARY_PATH="/opt/csw/lib:/usr/lib:${NH_ORA_CLIENT_HOME}/lib:/usr/dt/lib:${NH_HOME}/lib:${NH_HOME}/web/httpd/bin" 

export LD_LIBRARY_PATH

PATH=${NH_HOME}/bin:${NH_ORA_CLIENT_HOME}/bin:/usr/openwin/bin:/opt/csw/bin:/usr/sbin:/bin:/usr/bin:/etc:/usr/etc:${NhSavePath} 

 

2. Modified the below in "/export/home/neth/.bash_profile" file:

PATH=$PATH:/opt/csw/bin:$NH_HOME/custom/bin:$NH_HOME/bin/sys:$HOME/bin 

 

3. Found that there were 20 servers that had "/opt/csw/lib/libxml2.so.2" linked to "/opt/csw/lib/libxml2.so.2.9.1", which was causing errors with SSH in the environment. Copied over "/opt/csw/lib/libxml2.so.2.7.8" from a working server and linked it to "/opt/csw/lib/libxml2.so.2" to resolve the errors.

Additional Information:

NOTE: Placing the path to the new library at the start of the Path variable value triggered a failure of the nhServer process to start.