Reply
Expert Contributor
Posts: 136
Registered: ‎07-17-2017
Accepted Solution

Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Hello everyone,

Before upgrading CM and CDH from 5.10 to 5.12, I updated the apache kudu 1.3 to 1.4, but after the upgrading I have a communication problem of 4 tablet servers with the master server.

When I want to create a kudu table within Impala I has this message:

 

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_db.test2p'
CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a
table with the requested replication factor 3. 1 tablet servers are alive.

 


in kudu-master.ERROR file log of one of the tablets server I found this:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0718 10:50:33.369348 30017 master.cc:171] Master@0.0.0.0:7051: Unable to init 
master catalog manager: Timed out: Unable to initialize catalog manager: Failed
to initialize sys tables async: Failed to create new distributed Raft config:
Unable to resolve UUID for peer member_type: VOTER last_known_addr {
host: "nod7.exp.com" port: 7051 }: Getting permanent uuid from nod7.exp.com:7051
timed out after 30000 ms.: Network error: Client connection negotiation failed:
client connection to X.X.X.X:7051: connect: Connection refused (error 111)
F0718 10:50:33.369388 29904 master_main.cc:71] Check failed: _s.ok() Bad status:
Timed out: Unable to initialize catalog manager: Failed to initialize sys tables
async: Failed to create new distributed Raft config: Unable to resolve UUID for
peer member_type: VOTER last_known_addr { host: "nod7.exp.com" port: 7051 }:
Getting permanent uuid from nod7.exp.com:7051 timed out after 30000 ms.: Network
error: Client connection negotiation failed: client connection to X.X.X.X:7051:
connect: Connection refused (error 111)


and in kudu-tserver.WARNING :

W0718 15:17:18.787075 24931 heartbeater.cc:498] Failed to heartbeat to 
master.exp.com:7051: Not authorized: Failed to ping master at master.exp.com:7051
: Client connection negotiation failed: client connection to X.X.X.X:7051:
FATAL_UNAUTHORIZED: Not authorized: unauthenticated connections from publicly
routable IPs are prohibited. See --trusted_subnets flag for more information.:
X.X.X.X:41754


Hope we can resolve it without downgrading to kudu 1.3 :'(

Thanks in advance.

Cloudera Employee
Posts: 47
Registered: ‎02-05-2016

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Based on the contents of kudu-master.ERROR, it looks like your deployment has multiple masters, or perhaps had multiple masters in the past. Is that the case? I suspect that the error only surfaced after the upgrade because this may be the first time in a long while that you've restarted the master. Anyway, please describe the topology of your cluster.

Cloudera Employee
Posts: 30
Registered: ‎02-22-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Looking at the second error message, it seems to relate to a new security feature in 1.4 that blocks any unauthenticated or unauthorized connections coming from public ip. You may want to check the "trusted_subnets" flag description for detail to find out the suitable config of the flag for your deployment. If you need furthur clarification or help, please respond here.

Expert Contributor
Posts: 136
Registered: ‎07-17-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Thanks Adar for the quick answer,

In effect, you are right, after upgrading and when I faces the problems I had try to use 3 masters instead of one, but always no ssolution, so I return to 1 server architecture.

About the topology, I'm working with Impala under HDFS (1 namenode and 11 datanodes) and also under Kudu (1 master, 5 tablet servers).

The servers used in Kudu are the same used in HDFS.

Remark: Before updating kudu from 1.3 to 1.4 and CM/CDH from 5.10 to 5.12, I have no problems with the 2 structures.

Cloudera Employee
Posts: 47
Registered: ‎02-05-2016

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Going from 1 to 3 masters requires running a specific workflow. Based on what you wrote I'm not sure whether your single master is working now or not, but that definitely needs to be fixed first. What is the value of the --master_addresses command line parameter for your master?

 

Also, on the master machine, run the following code (adjusting the path as needed):

kudu pbc dump --oneline /var/lib/kudu/master/consensus-meta/00000000000000000000000000000000

 

There should be only one listed peer; if there is more than one, you'll need to use kudu local_replica cmeta rewrite_raft_config to fix it (see the workflow link above for details on how that command works).

Expert Contributor
Posts: 136
Registered: ‎07-17-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Hi Adar,

The --master_addresses command line parameter in a master is empty:

 

--master_addresses=


I run the kudu pbc dump --oneline.. code, and this is the result:

0	committed_config { opid_index: -1 OBSOLETE_local: true peers
{ permanent_uuid: "b5834d2fac62402799fae22b8da5d7b1" member_type: VOTER } }
current_term: 1 voted_for: "b5834d2fac62402799fae22b8da5d7b1"
Expert Contributor
Posts: 136
Registered: ‎07-17-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

[ Edited ]

Hi mister Hao Hao,

I find that really the "trusted_subnets" flag was the problem, I read about it, and how we can edit it and add a new authenticated address blocks. In the master I run this command line:

kudu master set_flag X.X.X.X --trusted_subnets 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,Y.Y.Y.Y/32 

but it can not be executed and return this error message:

Invalid argument: must provide flag

 


and when I use the tag without '--' :

kudu master set_flag X.X.X.X trusted_subnets 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,Y.Y.Y.Y/32 


I have this message:

Remote error: Not authorized: unauthorized access to method: SetFlag

 


& thanks a lot for you help.

 

 

Cloudera Employee
Posts: 30
Registered: ‎02-22-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

The error 'Remote error: Not authorized: unauthorized access to method: SetFlag' should indicate that you are using non-superuser client to set the flag. Try to set flag with superuser and see if it works.

Expert Contributor
Posts: 136
Registered: ‎07-17-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

In effect, I'm using a sudo user!
Highlighted
Expert Contributor
Posts: 136
Registered: ‎07-17-2017

Re: Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

Finaly I found the solution, in Kudu -> Configuration I modify this attribut

Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide)

by adding this line :

--trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24


Good luck.


Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.