About Juanes

Juanes · ‎05-29-2023

Good afternoon, I'm about to deploy CDP 7.1.8 and CP 7.7.1 but seems the licensing is going to take some time, my question is: is the Trial version same as licensed one? can I install Trial version and then update the cluster with the license? Question key: I'm about to build an HA environment, that means to deploy external Postgresql database, can I install Trial but to avoid using the embedded one? Many thanks in advance.

Juanes · ‎11-23-2022

Hello, I got the fix for this case, maybe this could help anyone having the same kudu Master consensus issue than me. Master1 is not voting: The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ Master1 A | A B C | 12026 | -1 | Yes Master2 B | A B C* | 12026 | -1 | Yes Master3 C | A B C* | 12026 | -1 | Yes the workarround is: A)stop the problematic Master and run the below command on Problematic master B)sudo -u kudu kudu local_replica delete --fs_wal_dir=/var/kudu/master --fs_data_dirs=/var/kudu/master 00000000000000000000000000000000 -clean_unsafe C) Please check the kudu leader master with webUI a98a1f26d0254293b6e17e9daf8f6ef8 822fcc68eff448269c9200a8c4c2ecc8 LEADER 2022-11-22 07:18:21 GMT rpc_addresses { host: "sdzw-hpas-35" port: 7051 } http_addresses { host: "sdzw-hpas-35" port: 8051 } software_version: "kudu 1.13.0.7.1.6.0-297 (rev 9323384dbd925202032a965e955979d6d2f6acb0)" https_enabled: false D)sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/wal/kudu/wal --fs_data_dirs=/wal/kudu/data 00000000000000000000000000000000 <active_leader_fqdn>:7051 # sudo -u kudu /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/bin/../lib/kudu/bin/kudu local_replica copy_from_remote --fs_wal_dir=/var/kudu/master --fs_data_dirs=/var/kudu/master 00000000000000000000000000000000 sdzw-hpas-35.nrtsz.local:7051 E)stop remaining two masters F)then start all the three masters.

Juanes · ‎11-14-2022

Hello, did you tried to use a Loadbalancer like HAproxy? I'm using Postgresql as HA internal database but for sure you can setup with both connections. something like this: frontend hive bind *:10000 mode tcp option tcplog timeout client 50000 default_backend hive_backend backend hive_backend mode tcp balance source timeout connect 5000 timeout server 50000 server hiveserver1 Master1:10000 server hiveserver2 Master2:10000

Juanes · ‎11-12-2022

Hello, In my 3Masters cluster, one Kudu Master is starting and stopping all the time, this is the Log detail from Cloudera Manager: Time Log Level Source Log Message 10:14:41.417 AM WARN cc:288 Found duplicates in --master_addresses: the unique set of addresses is Master1:7051, Master2:7051, Master3:7051 10:15:11.823 AM WARN cc:254 Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.55:55402 (request call id 0) took 4542 ms (4.54 s). Client timeout 1775 ms (1.78 s) 10:15:11.823 AM WARN cc:254 Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.37:59796 (request call id 0) took 30215 ms (30.2 s). Client timeout 9654 ms (9.65 s) 10:15:11.823 AM WARN cc:260 Trace: 1112 10:15:07.281146 (+ 0us) service_pool.cc:169] Inserting onto call queue 1112 10:15:07.281169 (+ 23us) service_pool.cc:228] Handling call 1112 10:15:11.823245 (+4542076us) inbound_call.cc:171] Queueing success response Metrics: {"spinlock_wait_cycles":384} 10:15:11.823 AM WARN cc:260 Trace: 1112 10:14:41.607787 (+ 0us) service_pool.cc:169] Inserting onto call queue 1112 10:14:41.607839 (+ 52us) service_pool.cc:228] Handling call 1112 10:15:11.823242 (+30215403us) inbound_call.cc:171] Queueing success response Metrics: {} 10:15:11.823 AM WARN cc:254 Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.55:55402 (request call id 1) took 4536 ms (4.54 s). Client timeout 1955 ms (1.96 s) 10:15:11.823 AM WARN cc:260 Trace: 1112 10:15:07.286988 (+ 0us) service_pool.cc:169] Inserting onto call queue 1112 10:15:07.287025 (+ 37us) service_pool.cc:228] Handling call 1112 10:15:11.823244 (+4536219us) inbound_call.cc:171] Queueing success response Metrics: {} What does it means??? why is this so unconsistent?

Juanes · ‎11-11-2022

Hello, In a CDP 7.1.6 + Cloudera Manager 7.3.1 cluster. 3Masters +3Workers. I'm getting the error all the time: Corruption: master consensus error: there are master consensus conflicts This is the cluster ksck: Master Summary UUID | Address | Status ----------------------------------+--------------------------+--------- 5620e4a103894151b7bdee5e436f37d8 | master-2.local | HEALTHY 9cea3b56cc9b4be4846a02c0d89be753 | master-1.local | HEALTHY a98a1f26d0254293b6e17e9daf8f6ef8 | master-3.local | HEALTHY All reported replicas are: A = 9cea3b56cc9b4be4846a02c0d89be753 B = 5620e4a103894151b7bdee5e436f37d8 C = a98a1f26d0254293b6e17e9daf8f6ef8 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ A | A B C | 10120 | -1 | Yes B | A B* C | 10120 | -1 | Yes C | A B* C | 10120 | -1 | Yes It seems the A node is not voting, this is the log output: W1111 11:12:00.526211 18688 leader_election.cc:334] T 00000000000000000000000000000000 P 9cea3b56cc9b4be4846a02c0d89be753 [CANDIDATE]: Term 10122 pre-election: RPC error from VoteRequest() call to peer 5620e4a103894151b7bdee5e436f37d8 (master-2:7051): Network error: Client connection negotiation failed: client connection to 10.157.136.55:7051: connect: Connection refused (error 111) W1111 11:12:22.683107 18688 leader_election.cc:334] T 00000000000000000000000000000000 P 9cea3b56cc9b4be4846a02c0d89be753 [CANDIDATE]: Term 10122 pre-election: RPC error from VoteRequest() call to peer 5620e4a103894151b7bdee5e436f37d8 (master-2:7051): Timed out: RequestConsensusVote RPC to 10.157.136.55:7051 timed out after 7.916s (SENT) there is conectivity: # nc -z -v 10.157.136.55 7051 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 10.157.136.55:7051. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. and the masters have been restarted several times, and the whole cluster... Any idea to fix this? Thanks!

Juanes · ‎08-25-2022

Added required dependencies to the repository: glibc-2.17-326.el7_9.i686 --> this was missing and that was why it tried to install 2.17-324 krb5-devel-1.15.1-50.el7.x86_64 openssl-devel-1.0.2k-21.el7_9.x86_64

Juanes · ‎08-24-2022

Yes, it should but not, i'm already using admin user from cloudera manager. So weird this all...

Juanes · ‎08-24-2022

Hello Scharan, tahnk you for your reply, Yes, I have read it thousand times but the problem is that the options (marked in RED) doen't exists in my Yarn Queue Manager... you can see it in the first image shown 😞

Juanes · ‎08-24-2022

Hello, it seems the main error is related to Impala, Kudu is balancing and responding well during the tests, the issue is that Impala breaks the connection to whoever that inform where the new Kudu Master LEADER is. I'm suspicious about the Cloudera Management services that are already down. Will update the solution whenever I have it.

Juanes · ‎08-24-2022

Many thanks for the recomendation, I will check again that it seems to be something related to the repositories, of course I will not downgrade glibc, this environment is quite important to broke it down 🙂

Online	Offline
Last Visited	‎09-13-2024 03:44 AM

Member Since	‎12-17-2020 04:51 AM
Last Visited	‎09-13-2024 03:44 AM
Posts	44
Kudos received	15

Cloudera Community

Re: Failed to install cloudera-manager-agent packa...

Re: DNS Resolution: hostname and canonical name fo...

Re: Failed to download https://archive.cloudera.co...

Re: Can't edit queues in Yarn Queue Manager / CM7....

Re: what recommended size of a kudu table

Limitations of CDP trial

Re: Unstable Kudu Master

Re: Configure HA database for Hive Metastore

Unstable Kudu Master

KUDU master consensus conflicts

Re: Cloudera Manager 7.3.1 installation error in R...

Re: Can't edit queues in Yarn Queue Manager / CM7....

Re: Can't edit queues in Yarn Queue Manager / CM7....

Re: ImpalaRuntimeException: Unable to initialize t...

Re: Cloudera Manager 7.3.1 installation error in R...