How to recover a corrupted MTP Vertica database.

Document ID : KB000012964
Last Modified Date : 14/02/2018
Show Technical Document Details
Question:

  How do I recover an improperly shut down MTP Vertica database ? (i.e. The database was not stopped before shutting down the MTP.)

Answer:

 For whatever reason, during the MTP shutdown, the Vertica database is suddenly closed. This may mean that it could get permanently damaged if not correctly brought up.  

If this is the case, follow these steps to correct the behavior.

 

 1.) Even though Vertica runs on localhost, we sometimes see problems when there are oddities/inconsistencies with the DNS resolution of the MTP host name. The best way usually to resolve this is to add an entry in the /etc/hosts file to make sure the MTP host name is properly resolved. 

a. On the MTP, add an entry to the /etc/hosts file for the MTP’s IP address and hostname. Note: you must have root privileges to edit this file; when logged     in using the netqos account, you will need to prefix the edit command with sudo:

    sudo vi /etc/hosts

    An example of the entry that gives the IP followed by the fully qualified domain name followed by the short name would be:
    10.10.0.12 mtp1.netqos.local mtp1

b. Confirm that the MTP’s IP address and hostname can now be resolved consistently by performing the following commands:
    hostname
    nslookup <ip address>
    nslookup <hostname>
    ping –a <hostname>


2.) Stop the processes that directly access the Vertica database.
sudo /opt/NetQoS/scripts/stopprocs.sh

3.) Confirm that there is no Vertica database processes running. To find whether a Vertica process is running, use the following command:
  ps –ef | grep vertica

  If the Vertica process is running, it will display a line similar to the following:
  dbadmin 9047 1 2 Jul02 ? 04:25:30 /opt/vertica/bin/vertica –C capture –D /nqxfs/vertica/capture/v_capture_node0001_catalog –h 127.0.0.1 –p 5433

If there is a process running, kill it using the Linux kill -9 <pid> command (e.g. kill -9 9047 would kill the above process).


4.) Manually drop the database (including making sure that the /nqxfs/vertica/capture folder is removed).

  su - dbadmin -c “/opt/vertica/bin/adminTools -t drop_db -d capture”
  (when prompted for password, enter ‘dbadmin’). Note syntax of command is different. It is ‘su dash’

  sudo rm -r /nqxfs/vertica/capture

5.) See Step 2

6) Restart the Vertica spreadd daemon.
  sudo /sbin/service spreadd stop
  sudo /sbin/service spreadd start

7.) Recreate the database.
  sudo /opt/NetQoS/install/setupVertica.sh –new

8.) Restart the MTP services
  sudo /opt/NetQoS/scripts/startprocs.sh

 

After doing all of the above, the MTP Vertica database should resume operations successfully.

 

Additional Information: