Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

GSS Initiate failed even with a valid kerberos service ticket

avatar
New Contributor

I have setup kerberos security on hadoop cluster using cloudera when i ran hdfs dfs -ls command it gives GSS initiate failed

I ran following commands

[

root@mac127 ~]# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local: addprinc -randkey hdfs
WARNING: no policy specified for hdfs@EXAMPLE.COM; defaulting to no policy
Principal "hdfs@EXAMPLE.COM" created.
kadmin.local: listprincs
HTTP/mac127.exmaple.com@EXAMPLE.COM
K/M@EXAMPLE.COM
cloudera-scm/admin@EXAMPLE.COM
hdfs/mac127.exmaple.com@EXAMPLE.COM
hdfs@EXAMPLE.COM
hive/mac127.exmaple.com@EXAMPLE.COM
host/mac127.exmaple.com@EXAMPLE.COM
hue/mac127.exmaple.com@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/mac127.exmaple.com@EXAMPLE.COM
kiprop/mac127.exmaple.com@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
mapred/mac127.exmaple.com@EXAMPLE.COM
oozie/mac127.exmaple.com@EXAMPLE.COM
root/admin@EXAMPLE.COM
yarn/mac127.exmaple.com@EXAMPLE.COM
zookeeper/mac127.exmaple.com@EXAMPLE.COM
kadmin.local: xst -norandkey -k /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type camellia256-cts-cmac added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type camellia128-cts-cmac added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
Entry for principal hdfs@EXAMPLE.COM with kvno 1, encryption type des-cbc-md5 added to keytab WRFILE:/etc/security/keytabs/hdfs.headless.keytab.
root@mac127 ~]# chown hdfs:hadoop /etc/security/keytabs/hdfs.headless.keytab
[root@mac127 ~]# chmod 440 /etc/security/keytabs/hdfs.headless.keytab
[root@mac127 ~]# su - hdfs
Last login: Fri Apr 28 11:11:42 IST 2017 on pts/1
-bash-4.2$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM

-bash-4.2$ klist
Ticket cache: FILE:/tmp/krb5cc_985
Default principal: hdfs@EXAMPLE.COM

Valid starting Expires Service principal
04/28/2017 11:14:51 04/29/2017 11:14:51 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 05/05/2017 11:14:51

 


later when I ran hdfs dfs -ls command i got below issue

 

-bash-4.2$ hdfs dfs -ls /
17/04/28 11:35:54 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
17/04/28 11:35:54 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
17/04/28 11:35:54 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1493359554217
17/04/28 11:35:58 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
17/04/28 11:35:58 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1493359554217
17/04/28 11:35:59 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
17/04/28 11:35:59 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1493359554217
17/04/28 11:36:02 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
17/04/28 11:36:02 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1493359554217
17/04/28 11:36:03 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
17/04/28 11:36:03 WARN ipc.Client: Couldn't setup connection for hdfs@EXAMPLE.COM to mac127.exmaple.com/172.27.155.127:8020
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561)
at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:376)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:731)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525)
at org.apache.hadoop.ipc.Client.call(Client.java:1448)
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2123)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1253)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
at org.apache.hadoop.fs.Globber.doGlob(Globber.java:285)
at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1703)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:102)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:372)
17/04/28 11:36:03 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for hdfs@EXAMPLE.COM to mac127.exmaple.com/172.27.155.127:8020
ls: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs@EXAMPLE.COM to mac127.exmaple.com/172.27.155.127:8020; Host Details : local host is: "mac127.exmaple.com/172.27.155.127"; destination host is: "mac127.exmaple.com":8020;
-bash-4.2$

could someone help me to get rid of this issue

Who agreed with this topic