How to recover a MTP Vertica database not correctly shut down.

Document ID : KB000013339
Last Modified Date : 14/02/2018
Show Technical Document Details
Question:

How do I recover an improperly shut down MTP Vertica database ? (i.e. The database not stopped before shutting down the MTP.)

Environment:
All MTP Releases.
Answer:

 

     When the MTP shuts down, the Vertica database is suddenly closed, So, it is likely that it may get permanently damaged if not correctly brought up.  

Follow these steps to correct the behavior.

 

  1. Even though Vertica runs on localhost, there can be problems when there are oddities/inconsistencies with DNS resolution of the MTP host name. The best way found to resolve this is to add an entry in the /etc/hosts file. This will make sure the MTP host name is properly resolved. 
 a. On the MTP, add an entry to the /etc/hosts file for the MTP’s IP address and hostname. Note: you must have root privileges to edit this file; when logged in using the netqos account, you will need to prefix the edit command with sudo:

sudo vi /etc/hosts

An example of the entry that gives the IP followed by the fully qualified domain name followed by the short name would be:
10.10.0.12 mtp1.netqos.local mtp1

b. Confirm that the MTP’s IP address and hostname can now be resolved consistently by performing the following commands:
hostname
nslookup <ip address>
nslookup <hostname>
ping –a <hostname>

 2. Stop the processes that directly access the Vertica database.
sudo /opt/NetQoS/scripts/stopprocs.sh

 3. Confirm that there are no Vertica database processes running. To find whether a Vertica process is running, use the following command:
ps –ef | grep vertica
If the Vertica process is running, it will display a line similar to the following:
dbadmin 9047 1 2 Jul02 ? 04:25:30 /opt/vertica/bin/vertica –C capture –D /nqxfs/vertica/capture/v_capture_node0001_catalog –h 127.0.0.1 –p 5433

If there is a process running, kill it using the Linux kill -9 <pid> command. (e.g. kill -9 9047 would kill the above process).


4. Manually drop the database (including making sure that the /nqxfs/vertica/capture folder is removed.)

su - dbadmin -c “/opt/vertica/bin/adminTools -t drop_db -d capture”
(when prompted for password, enter ‘dbadmin’). Note syntax of command is different. It is ‘su dash’

sudo rm -r /nqxfs/vertica/capture

5. See Step 2

6. Restart the Vertica spreadd daemon.
sudo /sbin/service spreadd stop
sudo /sbin/service spreadd start

7. Recreate the database.
sudo /opt/NetQoS/install/setupVertica.sh –new

8. Restart the MTP services
sudo /opt/NetQoS/scripts/startprocs.sh

 

      After doing the above, the MTP Vertica database should resume operations.

Additional Information:

PANIC - The MTP is not responding