Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datanodes can't authenticate to Namenode - kerberos

Solved Go to solution
Highlighted

Datanodes can't authenticate to Namenode - kerberos

Explorer

hello everyone,

please need help.

we have upgraded our cdh cluster from 5.16.2 to 6.3.2,

after upgrade looking at the logs, the datanodes cannot connect to the namenode using the existing kerberos principals

 

Datanodes logs:

    Caused by: java.lang.NullPointerException: Storage not yet initialized
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.hadoop.hdfs.server.datanode.DataNode.getVolumeInfo(DataNode.java:3178)
at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
... 35 more
2020-03-20 13:01:20,498 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: <namenode-fqdn>/<namenode-ip>:8022

 

namenode logs

 

 

org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled. Name node is in safe mode.
The reported blocks 0 needs additional 6820 blocks to reach the threshold 0.9990 of total blocks 6827.
The number of live datanodes 0 needs an additional 1 live datanodes to reach the minimum number 1.
Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:osscdh01.gre.hpecorp.net
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.newSafemodeException(FSNamesystem.java:1448)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1435)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4600)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:1276)
        at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:146)
        at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12974)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
2020-03-20 13:04:30,804 INFO org.apache.hadoop.ipc.Server: Connection from 172.17.98.94:27869 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user hdfs/hdfs/datanode-fqdn@REALM (auth:KERBEROS)
2020-03-20 13:04:30,953 INFO org.apache.hadoop.ipc.Server: Connection from 172.17.98.96:39391 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user hdfs/datanode-fqdn@REALM (auth:KERBEROS)

 

 

By the way is there any recommended kerberos version for cloudera 6.3 ?

 

has someone already encountered this issue ?

 

thanks 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Datanodes can't authenticate to Namenode - kerberos

Explorer

hi @Shelton  thank you for your reply.

yes this double hdfs is my error when trying to hide the principals.

Of course we regenerate keytab files, but the root cause was a hostname resolution problem.

We fixed the /etc/hosts file by switching from internal Ip addresses to external ones.

 

thank you

View solution in original post

3 REPLIES 3
Highlighted

Re: Datanodes can't authenticate to Namenode - kerberos

Mentor

@berti 

The first reflex is to regenerate the principals following the below steps

 

Review and regenerate the Kerberos principals for your cluster:

  1. Select Administration > Kerberos.
  2. The currently configured Kerberos principals are displayed. If you are running HDFS, the hdfs/hostname and host/hostname principals are listed. If you are running MapReduce, the mapred/hostname and host/hostname principals are listed. The principals for other running services are also listed.
  3. Only if necessary, select the principals you want to regenerate.
  4. Click Regenerate.

I have also noticed a discrepancy in the 2 principals I hope it was not a human error while you are trying to mask your real principals

  • user hdfs/hdfs/datanode-fqdn@REALM (auth:KERBEROS
  • user hdfs/datanode-fqdn@REALM (auth:KERBEROS)

This also appears in the logs

Connection from 172.17.98.94:27869 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user hdfs/hdfs/datanode-fqdn@REALM (auth:KERBEROS)

 

 

Connection from 172.17.98.96:39391 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user hdfs/datanode-fqdn@REALM (auth:KERBEROS)

 

Could you check that and revert.

 

 

 

Highlighted

Re: Datanodes can't authenticate to Namenode - kerberos

Explorer

hi @Shelton  thank you for your reply.

yes this double hdfs is my error when trying to hide the principals.

Of course we regenerate keytab files, but the root cause was a hostname resolution problem.

We fixed the /etc/hosts file by switching from internal Ip addresses to external ones.

 

thank you

View solution in original post

Re: Datanodes can't authenticate to Namenode - kerberos

Mentor

@berti 

 

Good to know it's resolved usually upgrading a cluster doesn't reset the hostnames in /etc/hosts? AWS internal IPs are for AWS inter datacenter communication  to expose the hosts you definitely need the public (external) IP's

 

Happy hadooping

 

Don't have an account?
Coming from Hortonworks? Activate your account here