Reply
Highlighted
Explorer
Posts: 7
Registered: ‎07-27-2015

Cloudera Kerberos setup : Login failure for hdfs/domain@REALM from hdfs.keytab: Connection refused

Hi,

 

We're trying to setup Kerberos on our cloudera cluster. We have managed to setup a working KDC and we even reached till the pen-ultimate step of cluster Kerberos setup wizard when Cloudera complained it couldn't start our services.

 

From what we can tell, the machines are discoverable to each other and we're able to do kinit successfully. 

 

Stack Trace:

 

Exception in secureMain
java.io.IOException: Login failure for hdfs/domain@realm from keytab hdfs.keytab: javax.security.auth.login.LoginException: Connection refused
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:976)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:243)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:207)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2288)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2337)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2514)
at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: javax.security.auth.login.LoginException: Connection refused
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:767)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:967)
... 11 more

 

 

Any help would be appreciated.

 

Explorer
Posts: 23
Registered: ‎02-21-2014

Re: Cloudera Kerberos setup : Login failure for hdfs/domain@REALM from hdfs.keytab: Connection refus

Hi,

 

I think that is not supposed to appear "hdfs/domain@realm", but hdfs/(hostname)@(SOMETHING.COM). Maybe, you've missed some step

in configuring HDFS with kerberos. A complete guide is at http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_sg_cdh5_hadoop_secu...

 

Sorry if not answered your request. Reply us here !!!

 

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Cloudera Kerberos setup : Login failure for hdfs/domain@REALM from hdfs.keytab: Connection refus

Does your KDC serve its port over both TCP and UDP? Could you check with
telnet and nc?
Explorer
Posts: 7
Registered: ‎07-27-2015

Re: Cloudera Kerberos setup : Login failure for hdfs/domain@REALM from hdfs.keytab: Connection refus

[ Edited ]

Thanks, that fixed the issue. I opened up both TCP and UDP ports and it's working now.

 

 

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Cloudera Kerberos setup : Login failure for hdfs/domain@REALM from hdfs.keytab: Connection refus

Its the Java JGSS API combined with the udp_preference_limit option in your
/etc/krb5.conf that requires TCP connectivity instead. Its better too
though, as the UDP mode has a message limit that often gets hit in a
growing environment.