Unable to access HDFS after enabling kerberos

[#1]

Hi,

I'm unable to access HDFS after enabling Kerberos authentication in my cluster. My namenode is in safe mode and is throwing out the below error when trying to execute leave safemode command.

PFB the error details:

 Failed to leave Safemode: at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy17.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:647)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy18.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2606)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1132)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1116)
at org.apache.hadoop.hdfs.tools.DFSAdmin.setSafeMode(DFSAdmin.java:571)
at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:1793)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:1967)
16/08/28 00:34:51 WARN security.UserGroupInformation: PriviledgedActionException as:hdfs/<hostname>@SPARTA.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for hdfs/<hostname>@SPARTA.COM to <hostname>/<ip-address>:8020
safemode: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs/<hostname>@SPARTA.COM to <hostname>/<ip-address>:8020; Host Details : local host is: "<hostname>/<ip-address>"; destination host is: "<hostname>":8020; 

Thanks,

Cibi

[#2]

Could you tail and check your NameNode log to observe what it prints as a security error when you attempt this request on it?

Does your command use the same JVM (with unlimited JCE jars installed if applicable) as the server does?

[#3 · Accepted solution]

Hi harsh,

The issue was with JCE files, i updated them in the wrong location instead of /usr/java/jdk1.7.0_25/jre/lib/security/.

Once i have updated the JCE files in the above location, now i'm able to access HDFS.

Thanks for the Help!

Regards,

Cibi