Tablet servers failed to heartbeat to master of Kudu 1.4.0 after upgrading from 1.3.0

[#1]

Hello everyone,

Before upgrading CM and CDH from 5.10 to 5.12, I updated the apache kudu 1.3 to 1.4, but after the upgrading I have a communication problem of 4 tablet servers with the master server.

When I want to create a kudu table within Impala I has this message:

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::kudu_db.test2p'
CAUSED BY: NonRecoverableException: Not enough live tablet servers to create a 
table with the requested replication factor 3. 1 tablet servers are alive.

in kudu-master.ERROR file log of one of the tablets server I found this:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0718 10:50:33.369348 30017 master.cc:171] Master@0.0.0.0:7051: Unable to init 
master catalog manager: Timed out: Unable to initialize catalog manager: Failed 
to initialize sys tables async: Failed to create new distributed Raft config: 
Unable to resolve UUID for peer member_type: VOTER last_known_addr { 
host: "nod7.exp.com" port: 7051 }: Getting permanent uuid from nod7.exp.com:7051 
timed out after 30000 ms.: Network error: Client connection negotiation failed: 
client connection to X.X.X.X:7051: connect: Connection refused (error 111)

F0718 10:50:33.369388 29904 master_main.cc:71] Check failed: _s.ok() Bad status: 
Timed out: Unable to initialize catalog manager: Failed to initialize sys tables 
async: Failed to create new distributed Raft config: Unable to resolve UUID for 
peer member_type: VOTER last_known_addr { host: "nod7.exp.com" port: 7051 }: 
Getting permanent uuid from nod7.exp.com:7051 timed out after 30000 ms.: Network 
error: Client connection negotiation failed: client connection to X.X.X.X:7051: 
connect: Connection refused (error 111)

and in kudu-tserver.WARNING :

W0718 15:17:18.787075 24931 heartbeater.cc:498] Failed to heartbeat to 
master.exp.com:7051: Not authorized: Failed to ping master at master.exp.com:7051
: Client connection negotiation failed: client connection to X.X.X.X:7051: 
FATAL_UNAUTHORIZED: Not authorized: unauthenticated connections from publicly 
routable IPs are prohibited. See --trusted_subnets flag for more information.: 
X.X.X.X:41754

Hope we can resolve it without downgrading to kudu 1.3 :'(

Thanks in advance.

[#2 · Accepted solution]

Finaly I found the solution, in Kudu -> Configuration I modify this attribut

Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile Kudu (Service-Wide)

by adding this line :

--trusted_subnets=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16,X.X.X.0/24

Good luck.

[#3]

What are the trusted subnets? I am also facing same issue in my cluster. If possible, can you please shed some light on this?