@Pritam Konar Is this issue resolved? Are you still facing this issue? ... View more

@Michael Bronson The Doc which i shared talks about Standard Best Practice. The doc does not say that running ZK on Kafka host will not work. But as a best practice you should keep them on separate hosts due to load constraints. However, it is subject to your Pre Prod Environment Testing and Metrics Analysis on both the scenarios and then you can proceed with what suite your requirement. ... View more

@Michael Bronson Kafka and Zookeepers are two services that are sensitive to disk I/O. Keeping them on same node will not be a good idea. The most performance-critical part of ZooKeeper is the transaction log. ZooKeeper must sync transactions to media before it returns a response. A dedicated transaction log device is key to consistent good performance. Putting the log on a busy device will adversely impact performance. The ZooKeeper transaction log must be configured in a dedicated device. This is very important to achieve best performance from ZooKeeper. ZooKeeper's transaction log must be on a dedicated device. ZooKeeper writes the log sequentially, without seeking. Sharing your log device with other processes can cause seeks and contention, which in turn can cause multi-second delays. ZooKeeper in a situation that can cause a swap. In order for ZooKeeper to function with any sort of timeliness, it simply cannot be allowed to swap. Remember, in ZooKeeper, everything is ordered, so if one request hits the disk, all other queued requests hit the disk. Some good discussions can be found on the following HCC threads / articles on this: 1. https://community.hortonworks.com/questions/55868/zookeeper-on-even-master-nodes.html 2. https://community.hortonworks.com/questions/2498/best-practices-for-zookeeper-placement.html 3. https://community.hortonworks.com/articles/62667/zookeeper-sizing-and-placement-draft.html ... View more

@Michael Bronson As per standard recommendation/ best practice: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_kafka-component-guide/content/kafka-zookeeper-multiple-apps.html Then for example: 1). Do not run ZooKeeper on a server where Kafka is running. If Kafka Brokers are installed on node1, node2, node3 then you should have Zookeepers on other cluster nodes where Kafka is not installed like node4, node5, node6. 2). When using ZooKeeper with Kafka you should dedicate ZooKeeper to Kafka, and not use ZooKeeper for any other components. Means the Zookeepers running on that node node4, node5, node6 should be used only for Kafka. Which means the Zookeeper running on those nodes (node4, node5, node6) should be dedicated to kafka means should not be used for other purpose like HBase/ NameNode Failovers /AMS ...etc ... View more

@Pritam Konar As we see the following error: Traceback (most recent call last):   File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 62, in _ensure_metadata     _group_entity = grp.getgrnam(group) KeyError: 'getgrnam(): name not found: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory\nroot' The above exception was the cause of the following exception: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Traceback (most recent call last):   File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SMARTSENSE/package/scripts/activity_analyzer.py", line 13, in <module>     Activity('analyzer').execute() So can you please try this and then see if it works for you? Edit the AGENT_WORKING_DIR property value inside the "/var/lib/ambari-agent/bin/ambari-agent" to something else and then try again. The default location will be something like following on all hosts. Default setting # grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent AGENT_WORKING_DIR=/var/lib/ambari-agent For example try changing it to something like "/tmp/ambari-agent " as following: Example After change # grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent AGENT_WORKING_DIR=/tmp/ambari-agent Then restart Ambari Agent on all hosts. # ambari-agent restart ... View more

@Pritam Konar As we see the following error: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Traceback (most recent call last):   File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 78, in <module>     AmsMonitor().execute() So can you please try this and then see if it works for you? Edit the AGENT_WORKING_DIR property value inside the "/var/lib/ambari-agent/bin/ambari-agent" to something else and then try again. The default location will be something like following on all hosts. Default setting # grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent AGENT_WORKING_DIR=/var/lib/ambari-agent For example try changing it to something like "/tmp/ambari-agent " as following: Example After change # grep 'AGENT_WORKING_DIR' /var/lib/ambari-agent/bin/ambari-agent AGENT_WORKING_DIR=/tmp/ambari-agent Then restart Amabri Agent on all hosts. # ambari-agent restart ... View more

@Spandan Mohanty Based on the recent error that you shared as "task-log-error.txt" we see the following failure: Access Denied (authentication_failed) Your credentials could not be authenticated: "General authentication failure due to bad user ID or authentication token.". You will not be permitted access until your credentials can be verified. This is typically caused by an incorrect username and/or password, but could also be caused by network problems. Possible Cause: This error indicates that there is some Network Proxy added to your Cluster and all your requests to nameNode ... and (may be other components) are going via Proxy Server and your proxy server is configured for authentication. So you will need to find the Proxy Settings added to any of the following places on your cluster hosts (including NameNode host and Ambari Server host) Possible Identify: You can find out if the requests are passing via Proxy or not if you try to run the same CRUL commands manually with "-iLv" options. # curl -iLv -X GET 'http://nodetwo:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs' Possible Remedy: Please search for "http_proxy" or "proxy" setting defined in your cluster nodes specially int he following files. (for "root" user and for "hdfs" user as well) and then remove them if needed or define the "no_proxy" As "root" user and "hdfs" or other users. # cat /etc/environment | grep proxy # cat ~/.bash_profile | grep proxy # cat /etc/profile | grep proxy You can disable the proxy setting for internal domain communication using "export no_proxy" option. To know more about please refer to : https://www.shellhacks.com/linux-proxy-server-settings-set-proxy-command-line/ ... View more

@luan ha Are you sure that "lunc-ht01" is your Public Hostname (OR) Private hostname ? Is that public hostname "lunc-ht01" mapped with IP Address "107.113.193.160" ? On the Nifi Host can you please try running the following command and share the output. # curl -s http://169.254.169.254/latest/meta-data/public-hostname -o /tmp/public-hostname.txt  ; cat /tmp/public-hostname.txt  # cat  /etc/hosts # hostname -f # netstat -tnlpa | grep 8080 . Can you check your Local machine where you are opening the web browser to verify your "/etc/hosts" file to see if it is pointing to the Private IP Address of "lunc-ht01" ? On your Local machine where you are opening the browser. # cat /etc/hosts # telnet 107.113.193.160 8080 (OR) # nc -v 107.113.193.160  8080 . ... View more