@Michael Bronson Kafka and Zookeepers are two services that are sensitive to disk I/O. Keeping them on same node will not be a good idea. The most performance-critical part of ZooKeeper is the transaction log. ZooKeeper must sync transactions to media before it returns a response. A dedicated transaction log device is key to consistent good performance. Putting the log on a busy device will adversely impact performance. The ZooKeeper transaction log must be configured in a dedicated device. This is very important to achieve best performance from ZooKeeper. ZooKeeper's transaction log must be on a dedicated device. ZooKeeper writes the log sequentially, without seeking. Sharing your log device with other processes can cause seeks and contention, which in turn can cause multi-second delays. ZooKeeper in a situation that can cause a swap. In order for ZooKeeper to function with any sort of timeliness, it simply cannot be allowed to swap. Remember, in ZooKeeper, everything is ordered, so if one request hits the disk, all other queued requests hit the disk. Some good discussions can be found on the following HCC threads / articles on this: 1. https://community.hortonworks.com/questions/55868/zookeeper-on-even-master-nodes.html 2. https://community.hortonworks.com/questions/2498/best-practices-for-zookeeper-placement.html 3. https://community.hortonworks.com/articles/62667/zookeeper-sizing-and-placement-draft.html ... View more

@Michael Bronson As per standard recommendation/ best practice: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-zookeeper-multiple-apps.html Then for example: 1). Do not run ZooKeeper on a server where Kafka is running. If Kafka Brokers are installed on node1, node2, node3 then you should have Zookeepers on other cluster nodes where Kafka is not installed like node4, node5, node6. 2). When using ZooKeeper with Kafka you should dedicate ZooKeeper to Kafka, and not use ZooKeeper for any other components. Means the Zookeepers running on that node node4, node5, node6 should be used only for Kafka. Which means the Zookeeper running on those nodes (node4, node5, node6) should be dedicated to kafka means should not be used for other purpose like HBase/ NameNode Failovers /AMS ...etc ... View more

@Pritam Konar As we see the following error: Traceback (most recent call last):   File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 62, in _ensure_metadata     _group_entity = grp.getgrnam(group) KeyError: 'getgrnam(): name not found: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory\nroot' The above exception was the cause of the following exception: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Traceback (most recent call last):   File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SMARTSENSE/package/scripts/activity_analyzer.py", line 13, in <module>     Activity('analyzer').execute() So can you please try this and then see if it works for you? Edit the AGENT_WORKING_DIR property value inside the "/var/lib/ambari-agent/bin/ambari-agent" to something else and then try again. The default location will be something like following on all hosts. Default setting # grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent AGENT_WORKING_DIR=/var/lib/ambari-agent For example try changing it to something like "/tmp/ambari-agent " as following: Example After change # grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent AGENT_WORKING_DIR=/tmp/ambari-agent Then restart Ambari Agent on all hosts. # ambari-agent restart ... View more

@Michael Bronson Zookeeper memory related settings can be specified inside the zookeeper-env script (via ambari zookeeper-env template) # grep 'SERVER_JVMFLAGS' /etc/zookeeper/3.1.0.0-78/0/zookeeper-env.sh export SERVER_JVMFLAGS=-Xmx4096m . ... View more

@Michael Bronson The meaning of "you should dedicate ZooKeeper to Kafka," and "Do not run ZooKeeper on a server where Kafka is running." statements are combined. Zookeeper Servers should not be installed/run on Kafka Broker host. Kafka should have zookeepers dedicated to it means the Zookeepers which are going to be by Kafka should not be used for other services. Like Kafka Zookeepers should not be used by HBase/ NameNode Failovers /AMS ...etc ... View more

@Naveen Varma Ramesh Can you please make sure that the "amb_ranger_admin" and ranger "admin" password are same. Please refer to the following links to see how to reset if needed. Please check the following UI paths: Ranger UI Ranger UI --> Settings --> Users/Groups Then Search for 'admin' and 'amb_ranger_admin' users. Ambari UI Ambari UI --> Ranger --> Configs --> Admin Settings --> "Ranger Admin user's password for Ambari"  Do you have the same password values that you used in Ranger UI ? For additional info please refer to : https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/authorization-ranger/content/ranger_install_updating_ranger_admin_passwords.html https://community.hortonworks.com/articles/186614/how-to-change-amb-ranger-admin-password.html . . ... View more

@Michael Bronson Do not run ZooKeeper on a server where Kafka is running. The statement is taken from the following doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-zookeeper-multiple-apps.html ... View more