https://community.cloudera.com/t5/Support-Questions/KUDU-master-consensus-conflicts/m-p/397550#M249895 <P>Hi Team as per the issue Master A is the fault one because once its in line there is no election happening please follow the below step to sync it you need to cleansafe the data from this master and replicate it back from leader master</P><P class="p1">A)stop the problematic Master</P><P class="p1">B)sudo -u kudu kudu local_replica delete --fs_wal_dir=/wal/kudu/wal --fs_data_dirs=/wal/kudu/data 00000000000000000000000000000000 -clean_unsafe</P><P class="p1">C)sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/wal/kudu/wal --fs_data_dirs=/wal/kudu/data 00000000000000000000000000000000 &lt;active_leader_fqdn&gt;:7051</P><P class="p1">D)stop remaining two masters</P><P class="p1">E)then start all the three masters.</P> Fri, 15 Nov 2024 10:07:07 GMT Asfahan 2024-11-15T10:07:07Z https://community.cloudera.com/t5/Support-Questions/KUDU-master-consensus-conflicts/m-p/357448#M237598 <P>Hello,</P><P>In a CDP 7.1.6 + Cloudera Manager 7.3.1 cluster.</P><P>3Masters +3Workers.</P><P>I'm getting the error all the time:&nbsp;Corruption: master consensus error: there are master consensus conflicts</P><P>&nbsp;</P><P>This is the cluster ksck:</P><P>Master Summary<BR />UUID | Address | Status<BR />----------------------------------+--------------------------+---------<BR />5620e4a103894151b7bdee5e436f37d8 | master-2.local | HEALTHY<BR />9cea3b56cc9b4be4846a02c0d89be753 | master-1.local | HEALTHY<BR />a98a1f26d0254293b6e17e9daf8f6ef8 | master-3.local | HEALTHY<BR />All reported replicas are:<BR />A = 9cea3b56cc9b4be4846a02c0d89be753<BR />B = 5620e4a103894151b7bdee5e436f37d8<BR />C = a98a1f26d0254293b6e17e9daf8f6ef8<BR />The consensus matrix is:<BR />Config source | Replicas | Current term | Config index | Committed?<BR />---------------+--------------+--------------+--------------+------------<BR />A | A B C | 10120 | -1 | Yes<BR />B | A B* C | 10120 | -1 | Yes<BR />C | A B* C | 10120 | -1 | Yes</P><P>&nbsp;</P><P>It seems the A node is not voting, this is the log output:<BR />W1111 11:12:00.526211 18688 leader_election.cc:334] T 00000000000000000000000000000000 P 9cea3b56cc9b4be4846a02c0d89be753 [CANDIDATE]: Term 10122 pre-election: RPC error from VoteRequest() call to peer 5620e4a103894151b7bdee5e436f37d8 (master-2:7051): Network error: Client connection negotiation failed: client connection to 10.157.136.55:7051: connect: Connection refused (error 111)<BR />W1111 11:12:22.683107 18688 leader_election.cc:334] T 00000000000000000000000000000000 P 9cea3b56cc9b4be4846a02c0d89be753 [CANDIDATE]: Term 10122 pre-election: RPC error from VoteRequest() call to peer 5620e4a103894151b7bdee5e436f37d8 (master-2:7051): Timed out: RequestConsensusVote RPC to 10.157.136.55:7051 timed out after 7.916s (SENT)</P><P>&nbsp;</P><P>there is conectivity:</P><P># nc -z -v 10.157.136.55 7051<BR />Ncat: Version 7.50 ( <A href="https://nmap.org/ncat" target="_blank" rel="noopener">https://nmap.org/ncat</A> )<BR />Ncat: Connected to 10.157.136.55:7051.<BR />Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.</P><P>and the masters have been restarted several times, and the whole cluster...</P><P>&nbsp;</P><P>Any idea to fix this? Thanks!</P> Fri, 11 Nov 2022 11:54:33 GMT https://community.cloudera.com/t5/Support-Questions/KUDU-master-consensus-conflicts/m-p/357448#M237598 Juanes 2022-11-11T11:54:33Z https://community.cloudera.com/t5/Support-Questions/KUDU-master-consensus-conflicts/m-p/397550#M249895 <P>Hi Team as per the issue Master A is the fault one because once its in line there is no election happening please follow the below step to sync it you need to cleansafe the data from this master and replicate it back from leader master</P><P class="p1">A)stop the problematic Master</P><P class="p1">B)sudo -u kudu kudu local_replica delete --fs_wal_dir=/wal/kudu/wal --fs_data_dirs=/wal/kudu/data 00000000000000000000000000000000 -clean_unsafe</P><P class="p1">C)sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/wal/kudu/wal --fs_data_dirs=/wal/kudu/data 00000000000000000000000000000000 &lt;active_leader_fqdn&gt;:7051</P><P class="p1">D)stop remaining two masters</P><P class="p1">E)then start all the three masters.</P> Fri, 15 Nov 2024 10:07:07 GMT https://community.cloudera.com/t5/Support-Questions/KUDU-master-consensus-conflicts/m-p/397550#M249895 Asfahan 2024-11-15T10:07:07Z