Vertica Outage

Document ID : KB000095261
Last Modified Date : 09/05/2018
Show Technical Document Details
Question:
What does "NETWORK change with 1 VS sets" mean when seen in a vertica.log just before we see "Node left cluster, reassessing k-safety"
Environment:
Any version of CAPM
Answer:
When you see:

018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw membership message 8192 (0x2000) on V:drdata 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw transitional message; watch for lost daemons 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw membership message 8192 (0x2000) on Vertica:all 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw transitional message; watch for lost daemons 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw membership message 8192 (0x2000) on Vertica:join 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw transitional message; watch for lost daemons 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> Saw membership message 6144 (0x1800) on V:drdata 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> NETWORK change with 1 VS sets 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> VS set #0 (mine) has 1 members (offset=36) 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> VS set #0, member 0: #node_a#N16807402424

... in the vertica.log just before seeing:

2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Comms] <INFO> nodeSetNotifier: node v_drdata_node0003 left the cluster 
2018-04-25 09:27:38.329 Spread Client:7fc55ecd2700 [Recover] <INFO> Node left cluster, reassessing k-safety... 

This is an indication of a network problem, possibly caused by slow disk speed, slow CPU processing speed, or slow network performance of the data repository hosts.  To check these things run the data repository diagnostic tools and make sure the vertica nodes meet the appropriate documented specifications
Additional Information:
Data Repository Diagnostic Utilities:
https://docops.ca.com/ca-performance-management/3-5/en/administrating/data-repository-administration/run-data-repository-diagnostic-utilities#RunDataRepositoryDiagnosticUtilities-vcpuperf