For whatever reason, during the MTP shutdown, the Vertica database is suddenly closed. This may mean that it could get permanently damaged if not correctly brought up.
If this is the case, follow these steps to correct the behavior.
1.) Even though Vertica runs on localhost, we sometimes see problems when there are oddities/inconsistencies with the DNS resolution of the MTP host name. The best way usually to resolve this is to add an entry in the /etc/hosts file to make sure the MTP host name is properly resolved.
a. On the MTP, add an entry to the /etc/hosts file for the MTP’s IP address and hostname. Note: you must have root privileges to edit this file; when logged in using the netqos account, you will need to prefix the edit command with sudo:
sudo vi /etc/hosts
An example of the entry that gives the IP followed by the fully qualified domain name followed by the short name would be:
10.10.0.12 mtp1.netqos.local mtp1
b. Confirm that the MTP’s IP address and hostname can now be resolved consistently by performing the following commands:
nslookup <ip address>
ping –a <hostname>
2.) Stop the processes that directly access the Vertica database.
3.) Confirm that there is no Vertica database processes running. To find whether a Vertica process is running, use the following command:
ps –ef | grep vertica
If the Vertica process is running, it will display a line similar to the following:
dbadmin 9047 1 2 Jul02 ? 04:25:30 /opt/vertica/bin/vertica –C capture –D /nqxfs/vertica/capture/v_capture_node0001_catalog –h 127.0.0.1 –p 5433
If there is a process running, kill it using the Linux kill -9 <pid> command (e.g. kill -9 9047 would kill the above process).
4.) Manually drop the database (including making sure that the /nqxfs/vertica/capture folder is removed).
su - dbadmin -c “/opt/vertica/bin/adminTools -t drop_db -d capture”
(when prompted for password, enter ‘dbadmin’). Note syntax of command is different. It is ‘su dash’
sudo rm -r /nqxfs/vertica/capture
5.) See Step 2
6) Restart the Vertica spreadd daemon.
sudo /sbin/service spreadd stop
sudo /sbin/service spreadd start
7.) Recreate the database.
sudo /opt/NetQoS/install/setupVertica.sh –new
8.) Restart the MTP services
After doing all of the above, the MTP Vertica database should resume operations successfully.