Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

[#1]

Hello everyone,

Before upgrading CM and CDH from 5.10 to 5.12, I updated the apache kudu 1.3 to 1.4, but after the upgrading I have a communication problem of 4 tablet servers with the master server.

When I want to create a kudu table within Impala I has this message:

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_db.test2p'
CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a 
table with the requested replication factor 3. 1 tablet servers are alive.

in kudu-master.ERROR file log of one of the tablets server I found this:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0718 10:50:33.369348 30017 master.cc:171] Master@0.0.0.0:7051: Unable to init 
master catalog manager: Timed out: Unable to initialize catalog manager: Failed 
to initialize sys tables async: Failed to create new distributed Raft config: 
Unable to resolve UUID for peer member_type: VOTER last_known_addr { 
host: "nod7.exp.com" port: 7051 }: Getting permanent uuid from nod7.exp.com:7051 
timed out after 30000 ms.: Network error: Client connection negotiation failed: 
client connection to X.X.X.X:7051: connect: Connection refused (error 111)

F0718 10:50:33.369388 29904 master_main.cc:71] Check failed: _s.ok() Bad status: 
Timed out: Unable to initialize catalog manager: Failed to initialize sys tables 
async: Failed to create new distributed Raft config: Unable to resolve UUID for 
peer member_type: VOTER last_known_addr { host: "nod7.exp.com" port: 7051 }: 
Getting permanent uuid from nod7.exp.com:7051 timed out after 30000 ms.: Network 
error: Client connection negotiation failed: client connection to X.X.X.X:7051: 
connect: Connection refused (error 111)

and in kudu-tserver.WARNING :

W0718 15:17:18.787075 24931 heartbeater.cc:498] Failed to heartbeat to 
master.exp.com:7051: Not authorized: Failed to ping master at master.exp.com:7051
: Client connection negotiation failed: client connection to X.X.X.X:7051: 
FATAL_UNAUTHORIZED: Not authorized: unauthenticated connections from publicly 
routable IPs are prohibited. See --trusted_subnets flag for more information.: 
X.X.X.X:41754

Hope we can resolve it without downgrading to kudu 1.3 :'(

Thanks in advance.

[#2]

Based on the contents of kudu-master.ERROR, it looks like your deployment has multiple masters, or perhaps had multiple masters in the past. Is that the case? I suspect that the error only surfaced after the upgrade because this may be the first time in a long while that you've restarted the master. Anyway, please describe the topology of your cluster.

[#3]

Thanks Adar for the quick answer,

In effect, you are right, after upgrading and when I faces the problems I had try to use 3 masters instead of one, but always no ssolution, so I return to 1 server architecture.

About the topology, I'm working with Impala under HDFS (1 namenode and 11 datanodes) and also under Kudu (1 master, 5 tablet servers).

The servers used in Kudu are the same used in HDFS.

Remark: Before updating kudu from 1.3 to 1.4 and CM/CDH from 5.10 to 5.12, I have no problems with the 2 structures.

[#4]

Going from 1 to 3 masters requires running a specific workflow. Based on what you wrote I'm not sure whether your single master is working now or not, but that definitely needs to be fixed first. What is the value of the --master_addresses command line parameter for your master?

Also, on the master machine, run the following code (adjusting the path as needed):

kudu pbc dump --oneline /var/lib/kudu/master/consensus-meta/00000000000000000000000000000000

There should be only one listed peer; if there is more than one, you'll need to use kudu local_replica cmeta rewrite_raft_config to fix it (see the workflow link above for details on how that command works).

[#5]

Hi Adar,

The --master_addresses command line parameter in a master is empty:

--master_addresses=

I run the kudu pbc dump --oneline.. code, and this is the result:

0	committed_config { opid_index: -1 OBSOLETE_local: true peers
{ permanent_uuid: "b5834d2fac62402799fae22b8da5d7b1" member_type: VOTER } } 
current_term: 1 voted_for: "b5834d2fac62402799fae22b8da5d7b1"

[#6]

Looking at the second error message, it seems to relate to a new security feature in 1.4 that blocks any unauthenticated or unauthorized connections coming from public ip. You may want to check the "trusted_subnets" flag description for detail to find out the suitable config of the flag for your deployment. If you need furthur clarification or help, please respond here.

[#7]

Hi mister Hao Hao,

I find that really the "trusted_subnets" flag was the problem, I read about it, and how we can edit it and add a new authenticated address blocks. In the master I run this command line:

kudu master set_flag X.X.X.X --trusted_subnets 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,Y.Y.Y.Y/32 

but it can not be executed and return this error message:

Invalid argument: must provide flag

and when I use the tag without '--' :

kudu master set_flag X.X.X.X trusted_subnets 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,Y.Y.Y.Y/32 

I have this message:

Remote error: Not authorized: unauthorized access to method: SetFlag

& thanks a lot for you help.

[#8]

The error 'Remote error: Not authorized: unauthorized access to method: SetFlag' should indicate that you are using non-superuser client to set the flag. Try to set flag with superuser and see if it works.

[#9]

Hi,

I am facing the same issue in my cluster. The trusted subnet flag is not configured with any value in my cluster. I have 3 masters. Please guide me what values to set in the trusted subnet flag.

[#10]

In effect, I'm using a sudo user!