Reply
Expert Contributor
Posts: 136
Registered: ‎07-17-2017

Communication Problem with master of Kudu 1.4.0-1.cdh5.12.0.p0.25 after upgrading from 1.3.0

[ Edited ]

Hi,

Before upgrading CM and CDH from 5.10 to 5.12.0-1.cdh5.12.0.p0.29, I updated the apache kudu 1.3 to 1.4.0-1.cdh5.12.0.p0.25, but after the upgrading I have a communication problem with 4 of 5 tablet server with the master.

When I want to create a kudu table within Impala I has this message:

 

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_db.test2p'
CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a
table with the requested replication factor 3. 1 tablet servers are alive.

 


in kudu-master.ERROR file log of one of the tablets server I found this:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0718 10:50:33.369348 30017 master.cc:171] Master@0.0.0.0:7051: Unable to init 
master catalog manager: Timed out: Unable to initialize catalog manager: Failed
to initialize sys tables async: Failed to create new distributed Raft config:
Unable to resolve UUID for peer member_type: VOTER last_known_addr {
host: "nod7.exp.com" port: 7051 }: Getting permanent uuid from nod7.exp.com:7051
timed out after 30000 ms.: Network error: Client connection negotiation failed:
client connection to X.X.X.X:7051: connect: Connection refused (error 111)
F0718 10:50:33.369388 29904 master_main.cc:71] Check failed: _s.ok() Bad status:
Timed out: Unable to initialize catalog manager: Failed to initialize sys tables
async: Failed to create new distributed Raft config: Unable to resolve UUID for
peer member_type: VOTER last_known_addr { host: "nod7.exp.com" port: 7051 }:
Getting permanent uuid from nod7.exp.com:7051 timed out after 30000 ms.: Network
error: Client connection negotiation failed: client connection to X.X.X.X:7051:
connect: Connection refused (error 111)


and in kudu-tserver.WARNING :

W0718 15:17:18.787075 24931 heartbeater.cc:498] Failed to heartbeat to 
master.exp.com:7051: Not authorized: Failed to ping master at master.exp.com:7051
: Client connection negotiation failed: client connection to X.X.X.X:7051:
FATAL_UNAUTHORIZED: Not authorized: unauthenticated connections from publicly
routable IPs are prohibited. See --trusted_subnets flag for more information.:
X.X.X.X:41754


I don't find any kind of documentation of this problem, hope can I resolve it without return to kudu 1.3 :'(
Hope you can hepl me ASAP.
Thanks in advance.