The following areas normally cause this problem: 1) the connection from Ambari agent host to Ambari Server got lost. 2) firewall issue blocked connections. 3) hostname and IP address are not being set correctly in /etc/hosts   You can compare the output using these APIs: > curl ­u user:paswd http://AmbariHost:8080/api/v1/hosts ... View more

Hi @mike_bronson7, Please follow this thread for the approach to handle such situations. https://community.cloudera.com/t5/Support-Questions/reinstall-ambari-or-add-existing-HDP-cluster-in-the-new/m-p/193982/highlight/true#M156042   Thank you. ... View more

@mike_bronson7    It seems to me like this is a symptom of having the default replication set to 3. This is for redundancy and processing capability within HDFS. It is recommended to have minimum 3 data nodes in the cluster to accommodate 3 healthy replicas of a block (as we have a default replication of 3).  HDFS will not write replicas of the same blocks to the same data node. In your scenario there will be under replicated blocks and 1 healthy replica will be placed on the available data node.   You may run setrep [1] to change the replication factor. If you provide a path to a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path.    hdfs dfs -setrep -w 1 /user/hadoop/dir1   [1] https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep ... View more

@mike_bronson7  Always stick to the Cloudera documentation. Yes !!! there is no risk in running that command I can understand your reservation.   ... View more

@mike_bronson7  Let me try to answer all your 3 questions in a shot [snapshot]   Zookeeper has 2 types of logs the snapshot and transactional log files. As changes are made to the znodes i.e addition or deletion of znodes these changes are appended to a transaction log, occasionally, when a log grows large, a snapshot of the current state of all znodes will be written to the filesystem. This snapshot supersedes all previous logs. To put you in context it's like the edit-logs and the fsimage in Namenode architecture, all changes made in the HDFS is logged in the edits-logs in secondary Namenode when a checkpoint kick in it merges the edits log with the old fsimage to incorporate the changes ever since the last checkpoint.   So zk snapshot is synonym to the fsimage as it contains the current state of the znode entries and ACL's Snapshot policy In the earlier command shared the snapshot count parameter -n <count> if you really want to have sleep then you can increment it to 5 or 7 but I think 3 suffice to use the autopurge feature so I keep only 3 snapshots and 3 transaction logs. When enabled, ZooKeeper auto-purge feature retains the autopurge.snapRetainCount most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 3. The minimum value is 3. Corrupt snapshots The Zookeeper might not be able to read its database and fail to come up because of some file corruption in the transaction logs of the ZooKeeper server.   You will see some IOException on loading the ZooKeeper database. In such a case, make sure all the other servers in your ensemble are up and working. Use the 4 letters command "stat" command on the command port to see if they are in good health. After you have verified that all the other servers of the ensemble are up, you can go ahead and clean the database of the corrupt server. Solution   Delete all the files in datadir/version-2 and datalogdir/version-2/. Restart the server. Hope that helps ... View more