Hello, did you tried to use a Loadbalancer like HAproxy? I'm using Postgresql as HA internal database but for sure you can setup with both connections. something like this: frontend hive bind *:10000 mode tcp option tcplog timeout client 50000 default_backend hive_backend backend hive_backend mode tcp balance source timeout connect 5000 timeout server 50000 server hiveserver1 Master1:10000 server hiveserver2 Master2:10000 ... View more

Yes, it should but not, i'm already using admin user from cloudera manager. So weird this all... ... View more

Hello Scharan, tahnk you for your reply, Yes, I have read it thousand times but the problem is that the options (marked in RED) doen't exists in my Yarn Queue Manager...  you can see it in the first image shown 😞   ... View more

Hello, it seems the main error is related to Impala, Kudu is balancing and responding well during the tests, the issue is that Impala breaks the connection to whoever that inform where the new Kudu Master LEADER is. I'm suspicious about the Cloudera Management services that are already down. Will update the solution whenever I have it. ... View more

Many thanks for the recomendation, I will check again that it seems to be something related to the repositories, of course I will not downgrade glibc, this environment is quite important to broke it down 🙂   ... View more

Hi, First of all, many thanks for your comments. I read about downgrading glibc could leave the system in undesired state :S  yu did downgraded glibc? I don't care about the kerberos packages, i'm not using it...   Best Regards ... View more

Hello, does anyone knows if exists any table reference with the errors? Just wanted to know what it means : Unable to initialize the Kudu scan node no relevant traces found in the following logs: Impala Daemon Impala Catalog Server Impala State Store Kudu Master Leader Kudu tablet Hive Metastore Hive Server2   I'm getting out of resources 😞   ... View more

Same problem here, 3 kudu masters, when 1 is down ...consensus problem. It's suposed that 2of3 should be majority to select the new leader. Note that the Leader is elected: uuid                             | rpc-addresses | role ---------------------------------+--------------+---------- daa9129e78245be2aaa7e5e649cc1dc8 | master1:7051 | LEADER 8a168b68a6dd587c952419672ba32088 | master3:7051 | FOLLOWER but queries against kudu is not working (ImpalaRuntimeException: Unable to initialize the Kudu scan node) ... 😕 ... View more

Hello , the ksck is showing that tables are OK ( Recovering | Under-replicated | Unavailable are all = 0 ) W0324 12:15:41.325619 18080 negotiation.cc:313] Failed RPC negotiation. Trace: 0324 12:15:40.627405 (+     0us) reactor.cc:609] Submitting negotiation task for client connection to master2:7051 0324 12:15:40.627616 (+   211us) negotiation.cc:98] Waiting for socket to connect 0324 12:15:41.325243 (+697627us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to master2:7051: connect: No route to host (error 113) Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":157,"threads_started":1} W0324 12:15:44.329090 18080 negotiation.cc:313] Failed RPC negotiation. Trace: 0324 12:15:41.325954 (+     0us) reactor.cc:609] Submitting negotiation task for client connection to master2:7051 0324 12:15:41.326027 (+    73us) negotiation.cc:98] Waiting for socket to connect 0324 12:15:44.329031 (+3003004us) negotiation.cc:304] Negotiation complete: Timed out: Client connection negotiation failed: client connection to master2:7051: Timeout exceeded waiting to connect Metrics: {"client-negotiator.queue_time_us":38} W0324 12:15:44.331089 18080 negotiation.cc:313] Failed RPC negotiation. Trace: 0324 12:15:44.329518 (+     0us) reactor.cc:609] Submitting negotiation task for client connection to master2:7051 0324 12:15:44.329580 (+    62us) negotiation.cc:98] Waiting for socket to connect 0324 12:15:44.331065 (+  1485us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to master2:7051: connect: No route to host (error 113) Metrics: {"client-negotiator.queue_time_us":36} Master Summary                   UUID                  |          Address           |   Status ----------------------------------------+----------------------------+-------------  8a168b68a6dd487c952419672ba32088       | master3.server.com | HEALTHY  daa9129e78244be2aaa7e5e649cc1dc8       | master1.server.com | HEALTHY   < unknown > (master2.server.com) | master2.server.com | UNAVAILABLE Error from master2.server.com: Network error: Client connection negotiation failed: client connection to master2:7051: connect: No route to host (error 113) (UNAVAILABLE) All reported replicas are:   A = daa9129e78244be2aaa7e5e649cc1dc8   B = < unknown > (master2.server.com)   C = 8a168b68a6dd487c952419672ba32088   D = 1f02c618009c44d381c55841dcb5a498 The consensus matrix is:  Config source |        Replicas        | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------  A             | A*      C   D          | 88           | -1           | Yes  B             | [config not available] |              |              |  C             | A*      C   D          | 88           | -1           | Yes Flags of checked categories for Master:         Flag         |                            Value                            |                         Master ---------------------+-------------------------------------------------------------+--------------------------------------------------------  builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | master1.server.com, master3.server.com  time_source         | system                                                      | master1.server.com, master3.server.com Tablet Server Summary                UUID               |             Address             | Status  | Location | Tablet Leaders | Active Scanners ----------------------------------+---------------------------------+---------+----------+----------------+-----------------  0dcf7da9a99c43dd99f956a95abb773d | worker3.server.com:7050 | HEALTHY | /default |      85        |       0  345166476e6440b4a957d76dd1de947e | worker2.server.com:7050 | HEALTHY | /default |     314        |       0  9e30289cb8d244e0a7fb822897fc7c29 | worker1.server.com:7050 | HEALTHY | /default |    1051        |       0 Tablet Server Location Summary  Location |  Count ----------+---------  /default |       3 Flags of checked categories for Tablet Server:         Flag         |                            Value                            |      Tablet Server ---------------------+-------------------------------------------------------------+-------------------------  builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked  time_source         | system                                                      | all 3 server(s) checked Version Summary       Version       |                                                               Servers --------------------+--------------------------------------------------------------------------------------------------------------------------------------  1.13.0.7.1.6.0-297 | master@master3.server.com, master@master1.server.com, tserver@worker3.server.com:7050, and 2 other server(s) Tablet Summary Summary by table                                   Name                                   | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Un available -------------------------------------------------------------------------+----+---------+- --------------+---------+------------+------------------+------------- Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1450 First Quartile | 1450 Median | 1450 Third Quartile | 1450 Maximum | 1450 Total Count Summary | Total Count ----------------+------------- Masters | 3 Tablet Servers | 3 Tables | 109 Tablets | 1450 Replicas | 4350 ================== Warnings: ================== master unusual flags check error: 1 of 3 masters were not available to retrieve unusual flags master diverged flags check error: 1 of 3 masters were not available to retrieve time_source category flags ================== Errors: ================== Network error: error fetching info from masters: failed to gather info from all masters: 1 of 3 had errors Corruption: master consensus error: there are master consensus conflicts   That I have no clear is why I'm having a consensus error if I have 2of3 Master UP and all 3 Tablet servers UP      Many thanks for your help. ... View more

Whow, this worked perfect! You make my day, my best wishes to you my friend!  😎 ... View more

Hello, This command is not working, the binaries are running under the parcel installation and they are not loaded in the systemctl. I've tried to execute the binaries from /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/kudu/sbin/kudu-tserver but it doesn't work properly.   root@server:~# sudo /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/kudu/sbin/kudu-tserver --help kudu-tserver: Warning: SetUsageMessage() never called Flags from ../../../../../src/kudu/cfile/block_cache.cc: -block_cache_capacity_mb (block cache capacity in MB) type: int64 default: 512 -block_cache_type (Which type of block cache to use for caching data. Valid choices are 'DRAM' or 'NVM'. DRAM, the default, caches data in regular memory. 'NVM' caches data in a memory-mapped file using the memkind library. To use 'NVM', libmemkind 1.8.0 or newer must be available on the system; otherwise Kudu will crash.) type: string default: "DRAM" -force_block_cache_capacity (Force Kudu to accept the block cache size, even if it is unsafe.) type: bool default: false [...] ... View more

Good morning and thanks for your help, I already read this page but the command wasn't loaded as systemctl proccess: # sudo service kudu-tserver restart Redirecting to /bin/systemctl restart kudu-tserver.service Failed to restart kudu-tserver.service: Unit not found. I checked the kudu-tablet proccess and it was executed from the /opt/parcel: kudu 30973 30965 99 07:47 ? 00:05:51 /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/kudu/sbin/kudu-tserver --tserver_master_addrs=master1,master2,master3 --flagfile=/var/run/cloudera-scm-agent/process/1546342994-kudu-KUDU_TSERVER/gflagfile   I tried as well executing this kudu-tserver but it doesn't work: # sudo /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/kudu/sbin/kudu-tserver stop usage: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/kudu/sbin/kudu-tserver ... View more