Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

avatar
Master Collaborator

Hello everyone,

Before upgrading CM and CDH from 5.10 to 5.12, I updated the apache kudu 1.3 to 1.4, but after the upgrading I have a communication problem of 4 tablet servers with the master server.

When I want to create a kudu table within Impala I has this message:

 

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_db.test2p'
CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a
table with the requested replication factor 3. 1 tablet servers are alive.

 


in kudu-master.ERROR file log of one of the tablets server I found this:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0718 10:50:33.369348 30017 master.cc:171] Master@0.0.0.0:7051: Unable to init 
master catalog manager: Timed out: Unable to initialize catalog manager: Failed
to initialize sys tables async: Failed to create new distributed Raft config:
Unable to resolve UUID for peer member_type: VOTER last_known_addr {
host: "nod7.exp.com" port: 7051 }: Getting permanent uuid from nod7.exp.com:7051
timed out after 30000 ms.: Network error: Client connection negotiation failed:
client connection to X.X.X.X:7051: connect: Connection refused (error 111)
F0718 10:50:33.369388 29904 master_main.cc:71] Check failed: _s.ok() Bad status:
Timed out: Unable to initialize catalog manager: Failed to initialize sys tables
async: Failed to create new distributed Raft config: Unable to resolve UUID for
peer member_type: VOTER last_known_addr { host: "nod7.exp.com" port: 7051 }:
Getting permanent uuid from nod7.exp.com:7051 timed out after 30000 ms.: Network
error: Client connection negotiation failed: client connection to X.X.X.X:7051:
connect: Connection refused (error 111)


and in kudu-tserver.WARNING :

W0718 15:17:18.787075 24931 heartbeater.cc:498] Failed to heartbeat to 
master.exp.com:7051: Not authorized: Failed to ping master at master.exp.com:7051
: Client connection negotiation failed: client connection to X.X.X.X:7051:
FATAL_UNAUTHORIZED: Not authorized: unauthenticated connections from publicly
routable IPs are prohibited. See --trusted_subnets flag for more information.:
X.X.X.X:41754


Hope we can resolve it without downgrading to kudu 1.3 :'(

Thanks in advance.

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Finaly I found the solution, in Kudu -> Configuration I modify this attribut

Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide)

by adding this line :

--trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24


Good luck.


View solution in original post

11 REPLIES 11

avatar
Master Collaborator

Finaly I found the solution, in Kudu -> Configuration I modify this attribut

Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide)

by adding this line :

--trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24


Good luck.


avatar
Explorer

What are the trusted subnets? I am also facing same issue in my cluster. If possible, can you please shed some light on this?