Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: The Cloudera Community will undergo maintenance on Saturday, August 17 at 12:00am PDT. See more info here.

Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

SOLVED Go to solution

Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor

Hello everyone,

Before upgrading CM and CDH from 5.10 to 5.12, I updated the apache kudu 1.3 to 1.4, but after the upgrading I have a communication problem of 4 tablet servers with the master server.

When I want to create a kudu table within Impala I has this message:

 

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_db.test2p'
CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a
table with the requested replication factor 3. 1 tablet servers are alive.

 


in kudu-master.ERROR file log of one of the tablets server I found this:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0718 10:50:33.369348 30017 master.cc:171] Master@0.0.0.0:7051: Unable to init 
master catalog manager: Timed out: Unable to initialize catalog manager: Failed
to initialize sys tables async: Failed to create new distributed Raft config:
Unable to resolve UUID for peer member_type: VOTER last_known_addr {
host: "nod7.exp.com" port: 7051 }: Getting permanent uuid from nod7.exp.com:7051
timed out after 30000 ms.: Network error: Client connection negotiation failed:
client connection to X.X.X.X:7051: connect: Connection refused (error 111)
F0718 10:50:33.369388 29904 master_main.cc:71] Check failed: _s.ok() Bad status:
Timed out: Unable to initialize catalog manager: Failed to initialize sys tables
async: Failed to create new distributed Raft config: Unable to resolve UUID for
peer member_type: VOTER last_known_addr { host: "nod7.exp.com" port: 7051 }:
Getting permanent uuid from nod7.exp.com:7051 timed out after 30000 ms.: Network
error: Client connection negotiation failed: client connection to X.X.X.X:7051:
connect: Connection refused (error 111)


and in kudu-tserver.WARNING :

W0718 15:17:18.787075 24931 heartbeater.cc:498] Failed to heartbeat to 
master.exp.com:7051: Not authorized: Failed to ping master at master.exp.com:7051
: Client connection negotiation failed: client connection to X.X.X.X:7051:
FATAL_UNAUTHORIZED: Not authorized: unauthenticated connections from publicly
routable IPs are prohibited. See --trusted_subnets flag for more information.:
X.X.X.X:41754


Hope we can resolve it without downgrading to kudu 1.3 :'(

Thanks in advance.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor

Finaly I found the solution, in Kudu -> Configuration I modify this attribut

Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide)

by adding this line :

--trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24


Good luck.


9 REPLIES 9

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Rising Star

Based on the contents of kudu-master.ERROR, it looks like your deployment has multiple masters, or perhaps had multiple masters in the past. Is that the case? I suspect that the error only surfaced after the upgrade because this may be the first time in a long while that you've restarted the master. Anyway, please describe the topology of your cluster.

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor

Thanks Adar for the quick answer,

In effect, you are right, after upgrading and when I faces the problems I had try to use 3 masters instead of one, but always no ssolution, so I return to 1 server architecture.

About the topology, I'm working with Impala under HDFS (1 namenode and 11 datanodes) and also under Kudu (1 master, 5 tablet servers).

The servers used in Kudu are the same used in HDFS.

Remark: Before updating kudu from 1.3 to 1.4 and CM/CDH from 5.10 to 5.12, I have no problems with the 2 structures.

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Rising Star

Going from 1 to 3 masters requires running a specific workflow. Based on what you wrote I'm not sure whether your single master is working now or not, but that definitely needs to be fixed first. What is the value of the --master_addresses command line parameter for your master?

 

Also, on the master machine, run the following code (adjusting the path as needed):

kudu pbc dump --oneline /var/lib/kudu/master/consensus-meta/00000000000000000000000000000000

 

There should be only one listed peer; if there is more than one, you'll need to use kudu local_replica cmeta rewrite_raft_config to fix it (see the workflow link above for details on how that command works).

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor

Hi Adar,

The --master_addresses command line parameter in a master is empty:

 

--master_addresses=


I run the kudu pbc dump --oneline.. code, and this is the result:

0	committed_config { opid_index: -1 OBSOLETE_local: true peers
{ permanent_uuid: "b5834d2fac62402799fae22b8da5d7b1" member_type: VOTER } }
current_term: 1 voted_for: "b5834d2fac62402799fae22b8da5d7b1"

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Contributor

Looking at the second error message, it seems to relate to a new security feature in 1.4 that blocks any unauthenticated or unauthorized connections coming from public ip. You may want to check the "trusted_subnets" flag description for detail to find out the suitable config of the flag for your deployment. If you need furthur clarification or help, please respond here.

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor

Hi mister Hao Hao,

I find that really the "trusted_subnets" flag was the problem, I read about it, and how we can edit it and add a new authenticated address blocks. In the master I run this command line:

kudu master set_flag X.X.X.X --trusted_subnets 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,Y.Y.Y.Y/32 

but it can not be executed and return this error message:

Invalid argument: must provide flag

 


and when I use the tag without '--' :

kudu master set_flag X.X.X.X trusted_subnets 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,Y.Y.Y.Y/32 


I have this message:

Remote error: Not authorized: unauthorized access to method: SetFlag

 


& thanks a lot for you help.

 

 

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Contributor

The error 'Remote error: Not authorized: unauthorized access to method: SetFlag' should indicate that you are using non-superuser client to set the flag. Try to set flag with superuser and see if it works.

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor
In effect, I'm using a sudo user!
Highlighted

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Expert Contributor

Finaly I found the solution, in Kudu -> Configuration I modify this attribut

Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide)

by adding this line :

--trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24


Good luck.