How to recover the Data Repository (Vertica) Database startup failed on one of the cluster node

Document ID : KB000010615
Last Modified Date : 14/02/2018
Show Technical Document Details
Introduction:

When starting Data Repository database, the following error appeared in dbLog.

*** Starting database: drdata *** 

Starting nodes: 

v_drdata_node0001 

v_drdata_node0002 

v_drdata_node0003 

Error: the vertica process for the database is running on the following hosts: <IP address> 

This may be because the process has not completed previous shutdown activities. Please wait and retry again. 

Database start up failed. Processes still running. 

Press RETURN to continue 

Environment:
OS: Red Hat Linux 6.x
Instructions:

This error would be caused by ungraceful previous shutdown.  Please try following (1).  If it is not still solved, try (2).

Note: 

  • The 'dradmin' account is the default database user.  Please use your database admin account if you changed it.
  • The adminTools can be executed from any node of Data Repository cluster.

 

(1) Simple method

# su - dradmin 

$ /opt/vertica/bin/adminTools 

Select Option "5 Restart Vertica on Host"

Select the problem node and click [OK]

Exit adminTools and dradmin user

$ exit 

 

(2) Forceful method

# su - dradmin 

$ /opt/vertica/bin/adminTools 

Select Option "7 Advanced Menu"

Select SubOption "3 Kill Vertica Process on Host"

Select the problem node and click [OK]

Exit adminTools and dradmin user

$ exit 

With root user run following command on all Data Repository boxes: 

# /etc/init.d/spreadd stop

Please confirm SPREAD process is not running with below.  

# ps -ef|grep spread 

Restart SPREAD process on all Data Repository boxes: 

# /etc/init.d/spreadd start 

Restart Data Repository process:  

# su - dradmin 

$ /opt/vertica/bin/adminTools 

Select Option "5 Restart Vertica on Host"

Select the problem node and click [OK]

Select Option "1 View Database Cluster State" to confirm all Data Repository nodes are UP state

Exit adminTools 

Additional Information:

You can also able to confirm the Data Repository status by following command:

# su - dradmin 

$ /opt/vertica/bin/adminTools -t list_allnodes