@Geoffrey shelton Okot, After refreshing the nodes we have kill all the processes and started from Ambari, Now all the services are up & running fine. Thank you so much for helping us on this.. ... View more

error-log1.pngerror-log2.png@Geoffrey, We have started the all the services successfully. datanode1 services are started but in ambari UI and hdfs admin report its not showing (showing it as dead node). We have Two Namenodes & Three data nodes. Error log and report log have been attached. report-3.txt ... View more

Latest log from datanode1: Connection failed: [Errno 111] Connection refused to rjio-prodcluster-datanode01:1019 ... View more

Hi Geoffrey, We have issued namenode recover command, now all the services are up and running fine ( with kerborose enabled). We have 3 data nodes and all the data nodes are up, now we are getting below alert on datanode1. Connection failed to http://rjio-prodcluster-datanode01:1022 (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -c /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -w '%{http_code}' http://rjio-prodcluster-datanode01:1022 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpcLoFod 2>/tmp/tmp7T5zRl' returned 7. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to rjio-prodcluster-datanode01:1022; Connection refused ... View more

Thanks for the reply Geoffrey Shelton, Did you install JCE to unlimited extension? --> We have installed jce_policy-8.zip #unzip -o -j -q <</fullpath>>/jce_policy-8.zip -d /usr/java/jdk1.8.0/jre/lib/security/ Have you tried restarting the cluster? --Yes Our hadoop cluster services were running fine without any issues, after enabling kerborose, we are not able to start the services. When we install kerborose at the last step it will try to restart all the cluster services and some of the services will not come up and operation will fail. ... View more