RHEL 7.3 Service Scripts not executing on reboot

Document ID : KB000074367
Last Modified Date : 21/03/2018
Show Technical Document Details
Introduction:
After installing NFA 9.3.8.24 on a test RHEL 7.3 for the Harvesters to prepare for a migration to it. It was noticed that the installer still creates the old service init scripts that are not used in RHEL 7.3 and doesn't use the new method.

This of course means the services do not start on reboot because RHEL 7 uses systemd now.
See https://access.redhat.com/solutions/912263 for information on how to convert old scripts to the new system.

The services can be manually started and everything works fine.
Background:
Verified that after reboot, when you do "service --status-all | grep NFA"  the services are stopped.

# service --status-all | grep -i nfa
NFA Collpollws is not running.
NFA Dataretention is not running.
NFA Filewebservice is not running.
NFA Harvester is not running.
NFA Poller is not running.
NFA Proxies is not running.
Unit nfa_reaper.service could not be found.

# rpm -qa | grep systemd
systemd-python-219-30.el7_3.7.x86_64
systemd-219-30.el7_3.7.x86_64
systemd-libs-219-30.el7_3.7.x86_64
systemd-sysv-219-30.el7_3.7.x86_64

After reboot they can manually be started and everything works fine. Just not after a reboot.

 
Environment:
Red Hat Enterprise Linux Server release 7.3 (Maipo)
RedHat 7.3 and above
RHEL 7.3 and above
NFA 9.x.x and above
Instructions:
Customer discovered systemd-sysv-generator helps systemd run sysV init scripts. Knowing this, he found these errors:

lnclnekvm200:~ # dmesg | grep generator
[    1.529901] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_mysqlCSE: No such file or directory
[    1.530548] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_harvester: No such file or directory
[    1.534633] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_dataretention: No such file or directory
[    1.534662] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_poller: No such file or directory
[    1.535557] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_collpollws: No such file or directory
[    1.535779] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_proxies: No such file or directory
[    1.535808] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_filewebservice: No such file or directory
[    1.535931] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_reaper: No such file or directory

Note the init scripts are symlinks:
lnclnekvm200:~ # ls -l /etc/rc.d/init.d
total 48
-rw-r--r--           1 root root  1160     Nov 29  2016 README
-rw-r--r--         1 root root 15131     Sep 12  2016 functions
-rwxrwxr-x      1 root root 10576     Mar  7 17:41 mysql
-rwxr-xr-x        1 root root  2989     Sep 12  2016 netconsole
-rwxr-xr-x        1 root root  6643     Sep 12  2016 network
lrwxrwxrwx    1 root root    41     Mar  7 17:44 nfa_collpollws -> /opt/CA/NFA/Netflow/bin/config_collpollws
lrwxrwxrwx    1 root root    44     Mar  7 17:44 nfa_dataretention -> /opt/CA/NFA/Netflow/bin/config_dataretention
lrwxrwxrwx     1 root root    45     Mar  7 17:44 nfa_filewebservice -> /opt/CA/NFA/Netflow/bin/config_filewebservice
lrwxrwxrwx     1 root root    40     Mar  7 17:44 nfa_harvester -> /opt/CA/NFA/Netflow/bin/config_harvester
lrwxrwxrwx     1 root root    39     Mar  7 17:42 nfa_mysqlCSE -> /opt/CA/NFA/Netflow/bin/config_mysqlCSE
lrwxrwxrwx     1 root root    37     Mar  7 17:44 nfa_poller -> /opt/CA/NFA/Netflow/bin/config_poller
lrwxrwxrwx     1 root root    38     Mar  7 17:44 nfa_proxies -> /opt/CA/NFA/Netflow/bin/config_proxies
lrwxrwxrwx     1 root root    40     Mar  7 17:44 nfa_reaper -> /opt/CA/NFA/Netflow/Reaper/config_reaper
-rwxr-xr-x     1 root root  2437     Jun 26  2015 rhnsd

The errors in dmesg suggested the symlink destination didn't exist -- which of course could happen during boot before the destination filesystem is mounted.
This seems to be by design per https://access.redhat.com/errata/RHBA-2016:0199: "Previously, systemd ignored symbolic links in the /etc/rc.d/init.d/ directory.
As a consequence, init scripts with symbolic links included in /etc/rc.d/init.d/ could not be started using systemd.
With this update, systemd follows symbolic links included in /etc/rc.d/init.d/ to init scripts.
Note that the real location of the init script must be on the partition that is mounted in the initial ramdisk (initrd). (BZ#1288005)"

Tried replacing the symlinks with copies and this worked:
cd /etc/rc.d/init.d

rm nfa_mysqlCSE
rm nfa_collpollws
rm nfa_dataretention
rm nfa_filewebservice
rm nfa_harvester
rm nfa_poller
rm nfa_proxies
rm nfa_reaper

cp -p /opt/CA/NFA/Netflow/bin/config_collpollws ./nfa_collpollws
cp -p /opt/CA/NFA/Netflow/bin/config_dataretention ./nfa_dataretention
cp -p /opt/CA/NFA/Netflow/bin/config_filewebservice ./nfa_filewebservice
cp -p /opt/CA/NFA/Netflow/bin/config_harvester ./nfa_harvester
cp -p /opt/CA/NFA/Netflow/bin/config_mysqlCSE ./nfa_mysqlCSE
cp -p /opt/CA/NFA/Netflow/bin/config_poller ./nfa_poller
cp -p /opt/CA/NFA/Netflow/bin/config_proxies ./nfa_proxies
cp -p /opt/CA/NFA/Netflow/Reaper/config_reaper ./nfa_reaper

/sbin/chkconfig --add nfa_mysqlCSE
/sbin/chkconfig --add nfa_collpollws
/sbin/chkconfig --add nfa_dataretention
/sbin/chkconfig --add nfa_filewebservice
/sbin/chkconfig --add nfa_harvester
/sbin/chkconfig --add nfa_poller
/sbin/chkconfig --add nfa_proxies
/sbin/chkconfig --add nfa_reaper

After a reboot, the services had started.

Whether you stick with this backward compatibility feature or implement actual systemd unit files,
you will probably see this problem unless the person installing happens to pick a filesystem that is mounted in initrd.
Additional Information:
The init script (or systemd unit) must be on the root (/) file system.
The systemd is the first daemon to run, and nothing but the root filesystem has been mounted at this point.
Only later in the boot process is the fstab processed.

As a quick test, the nfa_poller script was moved to the /var filesystem and symlinked and got the "stat() failed" message also and of course the service did not start.

Typical best practice is to not install the NFA applications to the root filesystem.