After installing NFA 220.127.116.11 on a test RHEL 7.3 for the Harvesters to prepare for a migration to it. It was noticed that the installer still creates the old service init scripts that are not used in RHEL 7.3 and doesn't use the new method.
This of course means the services do not start on reboot because RHEL 7 uses systemd now.
See https://access.redhat.com/solutions/912263 for information on how to convert old scripts to the new system.
The services can be manually started and everything works fine.
Verified that after reboot, when you do "service --status-all | grep NFA" the services are stopped.
# service --status-all | grep -i nfa
NFA Collpollws is not running.
NFA Dataretention is not running.
NFA Filewebservice is not running.
NFA Harvester is not running.
NFA Poller is not running.
NFA Proxies is not running.
Unit nfa_reaper.service could not be found.
# rpm -qa | grep systemd
After reboot they can manually be started and everything works fine. Just not after a reboot.
Red Hat Enterprise Linux Server release 7.3 (Maipo)
RedHat 7.3 and above
RHEL 7.3 and above
NFA 9.x.x and above
Customer discovered systemd-sysv-generator helps systemd run sysV init scripts. Knowing this, he found these errors:
lnclnekvm200:~ # dmesg | grep generator
[ 1.529901] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_mysqlCSE: No such file or directory
[ 1.530548] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_harvester: No such file or directory
[ 1.534633] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_dataretention: No such file or directory
[ 1.534662] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_poller: No such file or directory
[ 1.535557] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_collpollws: No such file or directory
[ 1.535779] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_proxies: No such file or directory
[ 1.535808] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_filewebservice: No such file or directory
[ 1.535931] systemd-sysv-generator: stat() failed on /etc/rc.d/init.d/nfa_reaper: No such file or directory
Note the init scripts are symlinks:
lnclnekvm200:~ # ls -l /etc/rc.d/init.d
-rw-r--r-- 1 root root 1160 Nov 29 2016 README
-rw-r--r-- 1 root root 15131 Sep 12 2016 functions
-rwxrwxr-x 1 root root 10576 Mar 7 17:41 mysql
-rwxr-xr-x 1 root root 2989 Sep 12 2016 netconsole
-rwxr-xr-x 1 root root 6643 Sep 12 2016 network
lrwxrwxrwx 1 root root 41 Mar 7 17:44 nfa_collpollws -> /opt/CA/NFA/Netflow/bin/config_collpollws
lrwxrwxrwx 1 root root 44 Mar 7 17:44 nfa_dataretention -> /opt/CA/NFA/Netflow/bin/config_dataretention
lrwxrwxrwx 1 root root 45 Mar 7 17:44 nfa_filewebservice -> /opt/CA/NFA/Netflow/bin/config_filewebservice
lrwxrwxrwx 1 root root 40 Mar 7 17:44 nfa_harvester -> /opt/CA/NFA/Netflow/bin/config_harvester
lrwxrwxrwx 1 root root 39 Mar 7 17:42 nfa_mysqlCSE -> /opt/CA/NFA/Netflow/bin/config_mysqlCSE
lrwxrwxrwx 1 root root 37 Mar 7 17:44 nfa_poller -> /opt/CA/NFA/Netflow/bin/config_poller
lrwxrwxrwx 1 root root 38 Mar 7 17:44 nfa_proxies -> /opt/CA/NFA/Netflow/bin/config_proxies
lrwxrwxrwx 1 root root 40 Mar 7 17:44 nfa_reaper -> /opt/CA/NFA/Netflow/Reaper/config_reaper
-rwxr-xr-x 1 root root 2437 Jun 26 2015 rhnsd
The errors in dmesg suggested the symlink destination didn't exist -- which of course could happen during boot before the destination filesystem is mounted.
This seems to be by design per https://access.redhat.com/errata/RHBA-2016:0199: "Previously, systemd ignored symbolic links in the /etc/rc.d/init.d/ directory.
As a consequence, init scripts with symbolic links included in /etc/rc.d/init.d/ could not be started using systemd.
With this update, systemd follows symbolic links included in /etc/rc.d/init.d/ to init scripts.
Note that the real location of the init script must be on the partition that is mounted in the initial ramdisk (initrd). (BZ#1288005)"
Tried replacing the symlinks with copies and this worked:
cp -p /opt/CA/NFA/Netflow/bin/config_collpollws ./nfa_collpollws
cp -p /opt/CA/NFA/Netflow/bin/config_dataretention ./nfa_dataretention
cp -p /opt/CA/NFA/Netflow/bin/config_filewebservice ./nfa_filewebservice
cp -p /opt/CA/NFA/Netflow/bin/config_harvester ./nfa_harvester
cp -p /opt/CA/NFA/Netflow/bin/config_mysqlCSE ./nfa_mysqlCSE
cp -p /opt/CA/NFA/Netflow/bin/config_poller ./nfa_poller
cp -p /opt/CA/NFA/Netflow/bin/config_proxies ./nfa_proxies
cp -p /opt/CA/NFA/Netflow/Reaper/config_reaper ./nfa_reaper
/sbin/chkconfig --add nfa_mysqlCSE
/sbin/chkconfig --add nfa_collpollws
/sbin/chkconfig --add nfa_dataretention
/sbin/chkconfig --add nfa_filewebservice
/sbin/chkconfig --add nfa_harvester
/sbin/chkconfig --add nfa_poller
/sbin/chkconfig --add nfa_proxies
/sbin/chkconfig --add nfa_reaper
After a reboot, the services had started.
Whether you stick with this backward compatibility feature or implement actual systemd unit files,
you will probably see this problem unless the person installing happens to pick a filesystem that is mounted in initrd.
The init script (or systemd unit) must be on the root (/) file system.
The systemd is the first daemon to run, and nothing but the root filesystem has been mounted at this point.
Only later in the boot process is the fstab processed.
As a quick test, the nfa_poller script was moved to the /var filesystem and symlinked and got the "stat() failed" message also and of course the service did not start.
Typical best practice is to not install the NFA applications to the root filesystem.