Kudu cluster - connection error

[#1]

We are trying to set up a three node kudu cluster as part of a POC. Our nodes are have additional ppk-layer of authentication. We have manually installed hdfs,yarn and impala successfully and are working fine. We have 3 kudu masters- one on each node - and 3 kudu tservers - one on each node. 

Contents of config files are :

/etc/default/kudu-master

export FLAGS_log_dir=/testpoc/hadoop_logs/kudu
export FLAGS_rpc_bind_addresses=0.0.0.0:7051

/etc/default/kudu-tserver

export FLAGS_log_dir=/testpoc/hadoop_logs/kudu
export FLAGS_rpc_bind_addresses=0.0.0.0:7050

/etc/kudu/conf/master.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-master.
--fromenv=rpc_bind_addresses
--fromenv=log_dir
--master_addresses=***.**.**.**,***.**.**.**,***.**.**.**

--fs_wal_dir=/testpoc/kudu/master
--fs_data_dirs=/testpoc/kudu/master

/etc/kudu/conf/tserver.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-tserver.
--fromenv=rpc_bind_addresses
--fromenv=log_dir
--master_addresses=***.**.**.**,***.**.**.**,***.**.**.**

--fs_wal_dir=/testpoc/kudu/tserver
--fs_data_dirs=/testpoc/kudu/tserver
--tserver_master_addrs=***.**.**.**,***.**.**.**,***.**.**.**

/etc/default/impala

IMPALA_CATALOG_SERVICE_HOST=***.**.**.**
IMPALA_STATE_STORE_HOST=***.**.**.**
IMPALA_STATE_STORE_PORT=24000
IMPALA_BACKEND_PORT=22000
IMPALA_LOG_DIR=/testpoc/hadoop_logs/impala
IMPALA_SCRATCH_DIR=/testpoc/impala_scratch_dir
KUDU_MASTER_HOSTS=***.**.**.**,***.**.**.**,***.**.**.**

IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} "
IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}"
IMPALA_SERVER_ARGS=" \
-log_dir=${IMPALA_LOG_DIR} \
-catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \
-state_store_port=${IMPALA_STATE_STORE_PORT} \
-use_statestore \
-state_store_host=${IMPALA_STATE_STORE_HOST} \
-be_port=${IMPALA_BACKEND_PORT} \
-scratch_dirs=${IMPALA_SCRATCH_DIR} \
-kudu_master_hosts=${KUDU_MASTER_HOSTS}"

ENABLE_CORE_DUMPS=false

kudu-master.ERROR

F0605 09:20:56.887897 23867 master_main.cc:74] Check failed: _s.ok() Bad status: Network error: error binding socket to ***.**.**.**:7051: Cannot assign requested address (error 99)

If we try to create a kudu table using impala shell, we get the following error:

ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::testdb5.kudutesttab1'
CAUSED BY: NonRecoverableException: can not complete before timeout: KuduRpc(method=ListTables, tablet=null, attempt=94, DeadlineTracker(timeout=180000, elapsed=176832), Traces: [0ms] querying master, [49ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [84ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [85ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [87ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [92ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [105ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [118ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [127ms] querying master, [127ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [127ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [128ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [129ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [134ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [143ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [147ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [167ms] querying master, [167ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [167ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [167ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [168ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [171ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [179ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [180ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [187ms] querying master, [187ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [187ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [187ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [188ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [203ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [223ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [234ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [267ms] querying master, [267ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [267ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [268ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [269ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [279ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [304ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [318ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [326ms] querying master, [326ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [326ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [327ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [327ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [338ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [342ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [348ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [407ms] querying master, [407ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [407ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [408ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [409ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [421ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [435ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [436ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [466ms] querying master, [466ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [467ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [467ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [468ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [470ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [471ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [473ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [567ms] querying master, [567ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [567ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [568ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [569ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [582ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [584ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [613ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [646ms] querying master, [647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [648ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [653ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [655ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [664ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [1626ms] querying master, [1626ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [1627ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [1627ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [1628ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [1629ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [1631ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [1632ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [1646ms] querying master, [1646ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [1647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [1647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [1648ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [1649ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [1650ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [1651ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [5326ms] querying master, [5327ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [5327ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [5329ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [5329ms] trace too long, truncated)
CAUSED BY: NoLeaderFoundException: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed
CAUSED BY: RecoverableException: connection closed

Are we missing something here? Let us know if more info is required.

Any kind of help is appreciated. Thanks.

[#2 · Accepted solution]

Seems it was ntp issue. Although, we did not see any clock synchronization related error.