@Spandan Mohanty The error indicates that your HDFS services are not healthy. Please check your NameNode logs. (/var/log/hadoop/hdfs/hadoop-hdfs-namenode-xxx.log) Because even the following simple webhdfs call is failing with 503 error. # curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' 'http://nodetwo:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs'' returned status_code=503. The cause of the error is logged as well here as following: The DNS server may be temporarily unavailable, or there could be a network problem. Next Action: 1. Please check if your HDFS services are up and running and there are no errors in NameNode logs. 2. Before attempting to start your "ResourceManagers" you should first check if the DNS name resolution is working fine and all your cluster nodes are able to resolve each other using their FQDN? Once your DNS issue is fixed and all the cluster nodes are able to resolve each other properly then restart the affected components and then try to validate if you are able to make the Simple WebHDFS call to name node or not ? Then only you should attempt to restart ResourceManager. # curl -sS -L -X GET 'http://nodetwo:50070/webhdfs/v1/ats/done/?op=GETFILESTATUS&user.name=hdfs' . ... View more

@Michael Bronson Zookeeper memory related settings can be specified inside the zookeeper-env script (via ambari zookeeper-env template) # grep 'SERVER_JVMFLAGS' /etc/zookeeper/3.1.0.0-78/0/zookeeper-env.sh export SERVER_JVMFLAGS=-Xmx4096m . ... View more

@Michael Bronson The meaning of "you should dedicate ZooKeeper to Kafka," and "Do not run ZooKeeper on a server where Kafka is running." statements are combined. Zookeeper Servers should not be installed/run on Kafka Broker host. Kafka should have zookeepers dedicated to it means the Zookeepers which are going to be by Kafka should not be used for other services. Like Kafka Zookeepers should not be used by HBase/ NameNode Failovers /AMS ...etc ... View more

@Naveen Varma Ramesh Can you please make sure that the "amb_ranger_admin" and ranger "admin" password are same. Please refer to the following links to see how to reset if needed. Please check the following UI paths: Ranger UI Ranger UI --> Settings --> Users/Groups Then Search for 'admin' and 'amb_ranger_admin' users. Ambari UI Ambari UI --> Ranger --> Configs --> Admin Settings --> "Ranger Admin user's password for Ambari"  Do you have the same password values that you used in Ranger UI ? For additional info please refer to : https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/authorization-ranger/content/ranger_install_updating_ranger_admin_passwords.html https://community.hortonworks.com/articles/186614/how-to-change-amb-ranger-admin-password.html . . ... View more

@Michael Bronson Do not run ZooKeeper on a server where Kafka is running. The statement is taken from the following doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-zookeeper-multiple-apps.html ... View more

@Anna Skobodzinski In HDFS, the blocks of the files are distributed among the datanodes as per the replication factor. If the Replication Factor is 1 then it means only on one DataNode the Data will be stored. Whenever you add a new datanode, the new datanodes will also start receiving and storing the blocks of the *New Files*. So just changing the replication factor will not cause rebalancing the cluster. Although the new files will be replicated based on the replication factor. What is HDFS Rebalancer? https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/data-storage/content/balancing_data_across_hdfs_cluster.html The HDFS Balancer is a tool for balancing the data across the storage devices of a HDFS cluster. The HDFS balancer moves blocks until the cluster is deemed to be balanced, which means that the utilization of every DataNode (ratio of used space on the node to total capacity of the node) differs from the utilization of the cluster (ratio of used space on the cluster to total capacity of the cluster) by no more than a given threshold percentage. The HDFS Rebalance operation can be either triggered via Ambari UI or via Command line: Ambari UI --> HDFS --> Actions (Drop down) --> Rebalance HDFS (Enter) then specify the "Balancer threshold (percentage of disk capacity)" Via Command line is described here: https://community.hortonworks.com/articles/43849/hdfs-balancer-2-configurations-cli-options.html https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/data-storage/content/balancing_data_across_hdfs_cluster.html You can check the "balancer.id" to see if it is going on (existing or not)? # su - hdfs -c "hdfs dfs -cat /system/balancer.id" Also you can check the "dfsadmin" report to find out the progress. # hdfs dfsadmin -report > before_dfsadmin.log . ... View more