We are trying to set up a three node kudu cluster as part of a POC. Our nodes are have additional ppk-layer of authentication. We have manually installed hdfs,yarn and impala successfully and are working fine. We have 3 kudu masters- one on each node - and 3 kudu tservers - one on each node.    Contents of config files are :   /etc/default/kudu-master export FLAGS_log_dir=/testpoc/hadoop_logs/kudu export FLAGS_rpc_bind_addresses=0.0.0.0:7051   /etc/default/kudu-tserver export FLAGS_log_dir=/testpoc/hadoop_logs/kudu export FLAGS_rpc_bind_addresses=0.0.0.0:7050     /etc/kudu/conf/master.gflagfile # Do not modify these two lines. If you wish to change these variables, # modify them in /etc/default/kudu-master. --fromenv=rpc_bind_addresses --fromenv=log_dir --master_addresses=***.**.**.**,***.**.**.**,***.**.**.** --fs_wal_dir=/testpoc/kudu/master --fs_data_dirs=/testpoc/kudu/master     /etc/kudu/conf/tserver.gflagfile # Do not modify these two lines. If you wish to change these variables, # modify them in /etc/default/kudu-tserver. --fromenv=rpc_bind_addresses --fromenv=log_dir --master_addresses=***.**.**.**,***.**.**.**,***.**.**.** --fs_wal_dir=/testpoc/kudu/tserver --fs_data_dirs=/testpoc/kudu/tserver --tserver_master_addrs=***.**.**.**,***.**.**.**,***.**.**.**   /etc/default/impala IMPALA_CATALOG_SERVICE_HOST=***.**.**.** IMPALA_STATE_STORE_HOST=***.**.**.** IMPALA_STATE_STORE_PORT=24000 IMPALA_BACKEND_PORT=22000 IMPALA_LOG_DIR=/testpoc/hadoop_logs/impala IMPALA_SCRATCH_DIR=/testpoc/impala_scratch_dir KUDU_MASTER_HOSTS=***.**.**.**,***.**.**.**,***.**.**.** IMPALA_CATALOG_ARGS=" -log_dir=${IMPALA_LOG_DIR} " IMPALA_STATE_STORE_ARGS=" -log_dir=${IMPALA_LOG_DIR} -state_store_port=${IMPALA_STATE_STORE_PORT}" IMPALA_SERVER_ARGS=" \ -log_dir=${IMPALA_LOG_DIR} \ -catalog_service_host=${IMPALA_CATALOG_SERVICE_HOST} \ -state_store_port=${IMPALA_STATE_STORE_PORT} \ -use_statestore \ -state_store_host=${IMPALA_STATE_STORE_HOST} \ -be_port=${IMPALA_BACKEND_PORT} \ -scratch_dirs=${IMPALA_SCRATCH_DIR} \ -kudu_master_hosts=${KUDU_MASTER_HOSTS}" ENABLE_CORE_DUMPS=false     kudu-master.ERROR   F0605 09:20:56.887897 23867 master_main.cc:74] Check failed: _s.ok() Bad status: Network error: error binding socket to ***.**.**.**:7051: Cannot assign requested address (error 99)     If we try to create a kudu table using impala shell, we get the following error:     ERROR: ImpalaRuntimeException: Error creating Kudu table 'impala::testdb5.kudutesttab1' CAUSED BY: NonRecoverableException: can not complete before timeout: KuduRpc(method=ListTables, tablet=null, attempt=94, DeadlineTracker(timeout=180000, elapsed=176832), Traces: [0ms] querying master, [49ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [84ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [85ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [87ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [92ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [105ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [118ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [127ms] querying master, [127ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [127ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [128ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [129ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [134ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [143ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [147ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [167ms] querying master, [167ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [167ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [167ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [168ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [171ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [179ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [180ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [187ms] querying master, [187ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [187ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [187ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [188ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [203ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [223ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [234ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [267ms] querying master, [267ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [267ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [268ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [269ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [279ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [304ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [318ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [326ms] querying master, [326ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [326ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [327ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [327ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [338ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [342ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [348ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [407ms] querying master, [407ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [407ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [408ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [409ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [421ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [435ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [436ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [466ms] querying master, [466ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [467ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [467ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [468ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [470ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [471ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [473ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [567ms] querying master, [567ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [567ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [568ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [569ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [582ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [584ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [613ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [646ms] querying master, [647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [648ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [653ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [655ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [664ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [1626ms] querying master, [1626ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [1627ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [1627ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [1628ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [1629ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [1631ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [1632ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [1646ms] querying master, [1646ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [1647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [1647ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA03:7051, [1648ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [1649ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA01:7051 response Network error: connection closed, [1650ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA03:7051 response Network error: connection closed, [1651ms] delaying RPC due to Service unavailable: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed, [5326ms] querying master, [5327ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA01:7051, [5327ms] Sub rpc: ConnectToMaster sending RPC to server master-TEST-POC-DATA02:7051, [5329ms] Sub rpc: ConnectToMaster received from server master-TEST-POC-DATA02:7051 response Network error: connection closed, [5329ms] trace too long, truncated) CAUSED BY: NoLeaderFoundException: Master config (TEST-POC-DATA01:7051,TEST-POC-DATA02:7051,TEST-POC-DATA03:7051) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed,org.apache.kudu.client.RecoverableException: connection closed CAUSED BY: RecoverableException: connection closed Are we missing something here? Let us know if more info is required. Any kind of help is appreciated. Thanks.       ... View more