When the MTP shuts down, the Vertica database is suddenly closed, So, it is likely that it may get permanently damaged if not correctly brought up.
Follow these steps to correct the behavior.
1. Even though Vertica runs on localhost, there can be problems when there are oddities/inconsistencies with DNS resolution of the MTP host name. The best way found to resolve this is to add an entry in the /etc/hosts file. This will make sure the MTP host name is properly resolved.
a. On the MTP, add an entry to the /etc/hosts file for the MTP’s IP address and hostname. Note: you must have root privileges to edit this file; when logged in using the netqos account, you will need to prefix the edit command with sudo:
sudo vi /etc/hosts
An example of the entry that gives the IP followed by the fully qualified domain name followed by the short name would be:
10.10.0.12 mtp1.netqos.local mtp1
b. Confirm that the MTP’s IP address and hostname can now be resolved consistently by performing the following commands:
nslookup <ip address>
ping –a <hostname>
2. Stop the processes that directly access the Vertica database.
3. Confirm that there are no Vertica database processes running. To find whether a Vertica process is running, use the following command:
ps –ef | grep vertica
If the Vertica process is running, it will display a line similar to the following:
dbadmin 9047 1 2 Jul02 ? 04:25:30 /opt/vertica/bin/vertica –C capture –D /nqxfs/vertica/capture/v_capture_node0001_catalog –h 127.0.0.1 –p 5433
If there is a process running, kill it using the Linux kill -9 <pid> command. (e.g. kill -9 9047 would kill the above process).
4. Manually drop the database (including making sure that the /nqxfs/vertica/capture folder is removed.)
su - dbadmin -c “/opt/vertica/bin/adminTools -t drop_db -d capture”
(when prompted for password, enter ‘dbadmin’). Note syntax of command is different. It is ‘su dash’
sudo rm -r /nqxfs/vertica/capture
5. See Step 2
6. Restart the Vertica spreadd daemon.
sudo /sbin/service spreadd stop
sudo /sbin/service spreadd start
7. Recreate the database.
sudo /opt/NetQoS/install/setupVertica.sh –new
8. Restart the MTP services
After doing the above, the MTP Vertica database should resume operations.