Support Questions

bibek_nep1234 · ‎09-19-2018

The Data Node fails to start with the following error:

ERROR datanode.DataNode (DataNode.java:secureMain(2692)) - Exception in secureMain
java.io.IOException: Login failure for dn/sandbox-hdp.hortonworks.com@HADOOP.COM from keytab /etc/security/keytabs/dn.service.keytab: javax.security.auth.login.LoginException: Receive timed out
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1098)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:307)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2491)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2540)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2685)
        at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: javax.security.auth.login.LoginException: Receive timed out
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1089)
        ... 10 more
Caused by: java.net.SocketTimeoutException: Receive timed out
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        at sun.security.krb5.internal.UDPClient.receive(NetClient.java:206)
        at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:411)
        at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:364)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.krb5.KdcComm.send(KdcComm.java:348)
        at sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:253)
        at sun.security.krb5.KdcComm.send(KdcComm.java:229)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:776)
        ... 23 more
2018-09-19 07:23:44,416 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2018-09-19 07:23:44,419 INFO  datanode.DataNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at sandbox-hdp.hortonworks.com/172.17.0.2

I implemented Kerberos from the Ambari UI. But after Kerberos is enbaled, I am not able to start the datanode. I checked on google for the error, found some solutions like:

1. Adding the following parameter to krb5.conf under the [libdefaults] section: udp_preference_limit = 1.

2 .Checking the hdfs-site.xml file which has entry :

<property> <name>dfs.datanode.kerberos.principal</name> <value>dn/_HOST@HADOOP.COM</value> </property> <property> <name>dfs.datanode.keytab.file</name> <value>/etc/security/keytabs/dn.service.keytab</value> </property>

3. Executing klint:

# kinit -V -k -t /etc/security/keytabs/dn.service.keytab dn/sandbox-hdp.hortonworks.com@HADOOP.COM

Using default cache: /tmp/krb5cc_0 Using principal: dn/sandbox-hdp.hortonworks.com@HADOOP.COM Using keytab: /etc/security/keytabs/dn.service.keytab

Authenticated to Kerberos v5

4. Result of cat /etc/hosts :

127.0.0.1 localhost

172.17.0.2 sandbox-hdp.hortonworks.com sandbox-hdp

127.0.0.1 sandbox.hortonworks.com

Do I need to do any additional things for kerberos apart from configuring through Ambari UI?

rlevas · ‎09-19-2018

@Bibek N

So I assume that there is a timeout error because the KDC is not responding in time. If the KDC was not found, we would see a different error. Where is the KDC located? If everything is one one host, maybe the host is overburdened and the KDC does not have cpu cycles to use.

Maybe take a look at the KDC logs to see if there is anything interesting - /var/log/krb5kdc.log and /var/log/kadmind.log.

Can you post your krb5.conf file?

bibek_nep1234 · ‎09-20-2018

@Robert Levas

Thanks for the reply. I found the problem and solved it. Even though I changed the protocol to TCP, it was trying to connect using UDP. KDC is in a different machine, and 88 port was open for only TCP connection, I changed it to allow both UDP and TCP and the problem got solved.