Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DataNode can't start after enabling Kerberos

DataNode can't start after enabling Kerberos

New Contributor

Hello, all!

After enabling Kerberos all services start fine except all DataNodes.

RHEL 7

HDP 2.5

krb5.conf

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = REALM.COM
 #default_ccache_name = KEYRING:persistent:%{uid}
[realms]
REALM.COM = {
 kdc = DC08
 admin_server = DC08
 }
[domain_realm]
 .realm.com = REALM.COM
 realm.com = REALM.COM

Erro in logfile

==> /var/log/hadoop/hdfs/jsvc.err <==
Initializing secure datanode resources
java.lang.IllegalArgumentException: Can't get Kerberos realm
 at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
 at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:289)
 at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:274)
 at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:351)
 at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:345)
 at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:92)
 at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.init(SecureDataNodeStarter.java:71)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:207)
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:88)
 at org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
 ... 11 more
Caused by: KrbException: Cannot locate default realm
 at sun.security.krb5.Config.getDefaultRealm(Config.java:1029)
 ... 17 more
Cannot load daemon
Service exit with a return value of 3
Initializing secure datanode resources
2 REPLIES 2

Re: DataNode can't start after enabling Kerberos

Rising Star

Hey, can you please verify the keytab exist for the datanode on the host (/etc/security/keytabs/)

Re: DataNode can't start after enabling Kerberos

@Nikita Kiselev

The problem may be the kdc and admin_server values. You have it set to "DC08". This needs to be the DNS name or IP address of the host where the KDC (or Active Directory) resides. Maybe you used "DC08" in place of the actual value to scrub the data before posting. If this is the case, make sure that all of the hosts in your cluster can get to that host.

If neither of those suggestions help, maybe there is a firewall in the way or maybe the port that the KDC is listening on has been changed. The default is 88. If this was changed, then you will want to append the port to the host value in kdc and admin_server properties. For example, if the port was changed to 9988, you would set the value to DC08:9988.