from your picture, it looks like you have a zookeeper on cmtoldhwdpadm01.dev.bmocm.com, how about cmtoldhwdpadm02, and cmtoldhwdpadm03? Are there zookeeper running on those two? Now, please follow the steps below: /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server cmtoldhwdpadm01.dev.bmocm.com ls /brokers/ids/ <-- here you should see two ids as there are two broker nodes get /brokers/ids/<id1> and <id2> <--- Please share the output ... View more

I think you have extra white spaces/line breaks at the end of file. ... View more

not sure what are the url for your zookeeper sever. you should run as /usr/hdp/current/zookeeper-server/bin/zkCli.sh -server cmtoldhwdpadm03.dev.bmocm.com:2181 if cmtoldhwdpadm03.dev.bmocm.com is your zookeeper server. ... View more

understand that your zookeeper is running. Want you to run zkCli at the command line, so that we can check whether it can connect from nodes to the zookeeper server. ... View more

try to check the zkCli, to see whether you are able to connect to zookeeper. As mentioned above post. ... View more

It looks like your brokers are not running correctly, or at least not registered on the zookeeper. Please check the kafka server log, and see what's the reason for that. The server log is located at /var/log/kafka. ... View more

you may consider Atlas and Ranger for that functionality. See this tutorial http://hortonworks.com/hadoop-tutorial/tag-based-policies-atlas-ranger/ for reference. ... View more

Yes. Datanode and nodemanager usually colocated. So, if you have 40 datanodes, then deploy 40 nodemanagers on these 40 datanodes. If you have some data that sit on the node that does not have nodemanager, then those data have to be transferred which increases the running time. ... View more

Very briefly looked over your original post, it seems that you sepearte Data nodes away from NodeManagers in your cluster B, which might increase the cost of data transferring among the nodes if the computing and data are not on the same node. In general, data node and node manager are colocated to guarantee the data locality as much as possible. I would suggest you try to set the cluster in that way, and see how the performance comes back. ... View more

If you specify "avro.compress=snappy" as TBLPROPERTIES, it will not work. You can try to set it in the command line: set hive.exec.compress.output=true; set avro.output.codec=snappy; and see whether it indeed compressed. ... View more