@Geoffrey Shelton Okot Running a ZK service check-Smoke test failed Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5/package/scripts/service_check.py", line 73, in <module> ZookeeperServiceCheck().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5/package/scripts/service_check.py", line 59, in service_check logoutput=True File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/zkSmoke.sh /usr/hdp/current/zookeeper-client/bin/zkCli.sh ambari-qa /usr/hdp/current/zookeeper-client/conf 2181 False kinit no_keytab no_principal /var/lib/ambari-agent/tmp/zkSmoke.out' returned 3. zk_node1=hdp.c.my-project-1519895027175.internal log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:708) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Running test on host hdp.c.my-project-1519895027175.internal Connecting to hdp.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: hdp.c.my-project-1519895027175.internal:2181(CONNECTING) 0] get /zk_smoketest Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:722) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Connecting to hdp.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: hdp.c.my-project-1519895027175.internal:2181(CONNECTING) 0] ls / Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for / at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:737) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:722) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Data associated with znode /zk_smoketests is not consistent on host hdp.c.my-project-1519895027175.internal Running test on host slave1.c.my-project-1519895027175.internal Connecting to slave1.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: slave1.c.my-project-1519895027175.internal:2181(CONNECTING) 0] get /zk_smoketest Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:722) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Connecting to slave1.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: slave1.c.my-project-1519895027175.internal:2181(CONNECTING) 0] ls / Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for / at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:737) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:722) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Data associated with znode /zk_smoketests is not consistent on host slave1.c.my-project-1519895027175.internal Running test on host slave2.c.my-project-1519895027175.internal Connecting to slave2.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: slave2.c.my-project-1519895027175.internal:2181(CONNECTING) 0] get /zk_smoketest Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:722) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Connecting to slave2.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: slave2.c.my-project-1519895027175.internal:2181(CONNECTING) 0] ls / Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for / at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:737) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1184) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:722) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Data associated with znode /zk_smoketests is not consistent on host slave2.c.my-project-1519895027175.internal Connecting to hdp.c.my-project-1519895027175.internal:2181 log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Welcome to ZooKeeper! JLine support is enabled [zk: hdp.c.my-project-1519895027175.internal:2181(CONNECTING) 0] delete /zk_smoketest Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /zk_smoketest at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:708) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:596) at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:368) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:328) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:287) Zookeeper Smoke Test: Failed ... View more

The really confusing part for me is Ambari shows ZK as running ... View more

@Geoffrey Shelton Okot I tried to connect to the Zookeeper server, but I'm getting the above error. (closing socket connection...) ... View more

1. I'm guessing my RM is already stopped 2. When I try to launch the Zookeeper Cli (./bin/zkCli.sh), I'm getting the following- Connecting to localhost:2181 2018-05-18 23:18:06,066 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 10:34 GMT 2018-05-18 23:18:06,068 - INFO [main:Environment@100] - Client environment:host.name=slave1.c.my-project-1519895027175.internal 2018-05-18 23:18:06,068 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_112 2018-05-18 23:18:06,070 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2018-05-18 23:18:06,070 - INFO [main:Environment@100] - Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre 2018-05-18 23:18:06,070 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/hdp/2.6.4.0-91/zookeeper/bin/....... . . . 2018-05-18 23:18:06,094 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) Welcome to ZooKeeper! . . . 2018-05-18 23:18:10,760 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-18 23:18:10,761 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@864] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:33932, server: localhost/0:0:0:0:0:0:0:1:2181 2018-05-18 23:18:10,761 - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1142] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect ... View more

Hmmm, when I try to start RM, I'm getting this- 2018-05-18 20:26:11,400 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZK operation. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:326) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:322) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1174) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1207) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDir(ZKRMStateStore.java:336) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDirRecursively(ZKRMStateStore.java:1311) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:303) at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:598) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:593) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1008) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1049) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1045) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1085) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1229) 2018-05-18 20:26:11,400 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1230)) - Retrying operation on ZK. Retry no. 203 2018-05-18 20:26:11,471 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server slave1.c.my-project-1519895027175.internal/10.142.0.3:2181. Will not attempt to authenticate using SASL (unknown error) 2018-05-18 20:26:11,472 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established, initiating session, client: /10.142.0.3:52748, server: slave1.c.my-project-1519895027175.internal/10.142.0.3:2181 2018-05-18 20:26:11,472 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect ... View more

@Geoffrey Shelton Okot Whole /etc/hosts file- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.142.0.2 hdp.c.my-project-1519895027175.internal hdp # Added by Google 169.254.169.254 metadata.google.internal # Added by Google 35.231.154.250 hdp.c.my-project-1519895027175.internal # Added by Mike Wong 35.231.170.209 slave1.c.my-project-1519895027175.internal #Added by Mike Wong 35.231.220.224 slave2.c.my-project-1519895027175.internal #Added by Mike Wong 35.229.111.57 slave3.c.my-project-1519895027175.internal #Added by Mike Wong ... View more

All of this look ok Firewalld status- ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) May 15 15:09:17 localhost systemd[1]: Starting firewalld - dynamic firewall daemon... May 15 15:09:19 localhost systemd[1]: Started firewalld - dynamic firewall daemon. May 15 21:03:24 slave3 systemd[1]: Stopping firewalld - dynamic firewall daemon... May 15 21:03:25 slave3 systemd[1]: Stopped firewalld - dynamic firewall daemon. route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.142.0.1 0.0.0.0 UG 100 0 0 eth0 10.142.0.1 0.0.0.0 255.255.255.255 UH 100 0 0 eth0 10.142.0.5 0.0.0.0 255.255.255.255 UH 100 0 0 eth0 ... View more

@Geoffrey Shelton Okot Cluster OS = RHEL7 VMs on GCP, four nodes total. Yes, SSH is working I can ping from each node to the other three nodes via IP address and FQDN successfully ... View more

@Sparsh, these four lines are present in the /etc/hosts file on all four of my nodes- 35.231.154.250 hdp.c.my-project-1519895027175.internal # Added by Mike Wong 35.231.170.209 slave1.c.my-project-1519895027175.internal #Added by Mike Wong 35.231.220.224 slave2.c.my-project-1519895027175.internal #Added by Mike Wong 35.229.111.57 slave3.c.my-project-1519895027175.internal #Added by Mike Wong getenforce resulted in Enforcing on three of my nodes. I've since disabled selinux using 'setenforce 0'. Now all four node are Permissive ... View more