Cloudera Community
nhgodwal
user rank
Contributor

NameNode not starting after enabling Kerberos on s...

by Contributor in Support Questions
‎10-15-2017 07:12 PM
‎10-15-2017 07:12 PM
I have setup single node HDP2.6 deployed using Ambari. My single node hostname is nhknox-Virtual-Machine.mad.lab I have KDC server setup at Domain Controller as Active Directory Domain Services. Now I am enabling Kerberos, which is going through fine. Then, on ambari wizard it stopping all the services, which is fine too. Then, it is trying to start all the services after kerberized hadoop. There my Namenode is not starting and below are the logs for the same. Please suggest for the solution stderr: stdout: 2017-10-15 13:58:14,561 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'} 2017-10-15 13:58:18,931 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-testknox@MAD.LAB'] {'user': 'hdfs'} 2017-10-15 13:58:19,157 - Waiting for this NameNode to leave Safemode due to the following conditions: HA: False, isActive: True, upgradeType: None 2017-10-15 13:58:19,157 - Waiting up to 19 minutes for the NameNode to leave Safemode... 2017-10-15 13:58:19,158 - Execute['/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://nhknox-virtual-machine.mad.lab:8020 -safemode get | grep 'Safe mode is OFF''] {'logoutput': True, 'tries': 115, 'user': 'hdfs', 'try_sleep': 10} safemode: Call From nhknox-virtual-machine.mad.lab/127.0.1.1 to nhknox-virtual-machine.mad.lab:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2017-10-15 13:58:44,683 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://nhknox-virtual-machine.mad.lab:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. safemode: Call From nhknox-virtual-machine.mad.lab/127.0.1.1 to nhknox-virtual-machine.mad.lab:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 2017-10-15 13:59:16,062 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://nhknox-virtual-machine.mad.lab:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 2017-10-15 13:59:45,987 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://nhknox-virtual-machine.mad.lab:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. 2017-10-15 14:00:15,016 - Retrying after 10 seconds. Reason: Execution of '/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs hdfs://nhknox-virtual-machine.mad.lab:8020 -safemode get | grep 'Safe mode is OFF'' returned 1. ... View more
Labels:

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 05:28 PM
‎10-13-2017 05:28 PM
@Aditya Sirna Thanks for helping me out in this case! But I am little confused that Why I need to create admin principal. Please read below for what I am trying to do and how my current setup is. Actually, my intent is to use existing Active Directory. I think, I need to create admin principal when I intent to use Existing MIT KDC, which is not the case. Please comment if I am wrong. (Also, do i need to be running kdc-admin-server and krb5kdc services if I am not using Existing MIT KDC?) Brief desc about KDC and hadoop setup: My existing Active Directory is already setup at Domain Controller (let's say, hostname=DC1) . And there I created new container "KnoxUsers" and a admin user "knxadmin" in that container. I want to use KDC as DC1 to connect to Active Directory from my hostmachine(where single node hadoop cluster is deployed using Ambari). To do that I started with enabling Kerberos through Ambari. After entering Configs (via Ambari) for KDC and Kadmin, it automatically started installing Kerberos client (which is succesfully complete, then it started Test Kerberos Client, where it is failed with the error "Failed to create principal - hadoop@domain - can not check if principal exists" ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 05:05 PM
‎10-13-2017 05:05 PM
@Geoffrey Actually, my intent is to use existing Active Directory. I think, document you pointed is when I intent to use Existing MIT KDC, which is not the case. My existing Active Directory is setup already at Domain Controller (let's say, hostname=DC1) . And there I created new container and a admin user in that container. I want to use KDC as DC1 to connect to Active Directory from my hostmachine(where single node hadoop cluster is deployed using Ambari). Do you think the steps you mentioned to setup Kerberos configs (before enabling Kerberos) are aligned with my case? ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 04:38 PM
‎10-13-2017 04:38 PM
@Geoffrey Thank you so much, for detailed steps! But before moving further, I have few questions (you might find dumb but would be great if you can help since I am a complete newbie): 1) My assumption was that the my KDC server is Domain Controller(windows server 2012) where my Active Directory Domain Services are running. In this case Why do I need to install KDC server on my ubuntu host machine where my hadoop is running? 2) Why do I need to create admin on my host machine? I thought the user I created on my Active Directory on DC i.e. knxadmin can be used to login as admin. 3) I created KnoxUser container and knxadmin user in that container by following below guideline: https://www.ibm.com/support/knowledgecenter/SSPT3X_4.2.0/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_kerb_activedir.html I hope, its a right way to do that? ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 06:47 AM
‎10-13-2017 06:47 AM
@tsharma Still same error as I tried removing all the lines accept the last one. Also, kadmin.local command is giving me error : Authenticating as principat root/admin@mydomain with password. kadmin.local: no such file or directory while initializing kadmin.local interface ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 06:35 AM
‎10-13-2017 06:35 AM
@Aditya Sirna I tried to create admin principal but it gave me error: Authenticating as principat root/admin@mydomain with password. kadmin.local: no such file or directory while initializing kadmin.local interface ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 06:32 AM
‎10-13-2017 06:32 AM
@tsharma I created kadm5.acl file with below lines - changed Example.COM to my domain: /admin@ATHENA.MIT.EDU * joeadmin@ATHENA.MIT.EDU ADMCIL joeadmin/*@ATHENA.MIT.EDU il */root@ATHENA.MIT.EDU */root@ATHENA.MIT.EDU cil *1@ATHENA.MIT.EDU */*@ATHENA.MIT.EDU i */admin@EXAMPLE.COM x * -maxlife 9h -postdateable I restarted krb5-admin-server service. In kadmin config, I changed Admin_principal to knxadmin/admin@MYDOMAIN but that didn't work. Still getting same error. ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 05:35 AM
‎10-13-2017 05:35 AM
I installed krb5-kdc krb5-admin-server but didn't find kadm5.acl in /etc/krb5kdc. I can see /etc/krb5kdc/kdc.conf having acl_file = /etc/krb5kdc/kadm5.acl. But there is no .acl file present in this directory. Should I create one? ... View more

Re: Failed to create principal - hadoop@domain - c...

by Contributor in Support Questions
‎10-13-2017 04:54 AM
‎10-13-2017 04:54 AM
@tsharma I didn't find /etc/krb5kdc Do I need to create this acl file? Note: [My host is ubuntu where single node hadoop is deployed using Ambari. ] ... View more

Failed to create principal - hadoop@domain - can n...

by Contributor in Support Questions
‎10-13-2017 03:07 AM
‎10-13-2017 03:07 AM
I am trying to ENABLE Kerberos. I entered KDC configs and then Kadmin_host, admin_principal and admin_password. Then it is successfully installing Kerberos cliend but failing while testing Kerberos client with below error: Failed to create principal - hadoop@domain - can not check if principal exists -> I find this error in "ambari-audit.log" I have a container in my Active Directory called KnoxUsers and I have "knxadmin" as user. I have put 'knxadmin' as Admin_Principal while configuring Kerberos. Can anyone points me where I need to check to correct this error? Brief Description about my cluster: I have single node cluster which I deployed using ambari on a VM on a host-machine. I have active directory setup on a domain controller. My domain controller and host-machine are on the same domain. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎10-18-2018 10:31 PM
Community Statistics
Member Since ‎10-03-2017 09:03 PM
Last Visited ‎10-18-2018 10:31 PM
Posts 82
Kudos received 2