Datanode can't connect to the namenode after enabling kerberos

[#1]

Hello Everyone,

I've recently enabled Kerberos in my cluster and since then one of my datanodes is not able to connect to the namenode:

I see these entries in the namenode logs:

2021-02-03 12:06:15,699 INFO  ipc.Server (Server.java:saslProcess(1573)) - Auth successful for $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO  ipc.Server (Server.java:authorizeConnection(2235)) - Connection from <datanode-ip>:42328 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/<datanode-hostname>@<Realm-Name> (auth:PROXY) via $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO  ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client <datanode-ip> threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $4E8100-MH1MCLUV65LO@<Realm-Name> is not allowed to impersonate dn/<datanode-hostname>@<Realm-Name>]

From the datanode logs:

2021-02-03 12:14:33,806 WARN  datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(225)) - Problem connecting to server: <namenode-hostname>/<namenode-ip>:8020

I've ensured that the hostnames are in lowercase and consistent in all the nodes.

Telnet is also happening from the datanode to namenode hostname on 8020 port.

Regenerating Keytabs and restarting everything also didn't work.

Any other areas to look into?

Thanks,

Megh

[#2 · Accepted solution]

@vidanimegh what java version you are using? This seems you are hitting a well known issue with java.  Cloudera product releases using one of the following JDK versions:

 OpenJDK 1.8 u242 and later updates(example u252, u262 etc)
 OpenJDK 11.0.6 and later updates
 Oracle JDK 8u241 and later updates

are affected by this. Below are the instructions to mitigate this issue.  

Instructions:

On all impacted nodes, this issue has been solved pursuant to implementing the steps below based on the environment. Edit the java.security file located in the active JDK on each node. You can find this file at one of the following locations:

 <JAVA_HOME>/jre/lib/security/
 <JAVA_HOME>/conf/security 

Add or alter the sun.security.krb5.disableReferrals parameter so that it is set to true:

sun.security.krb5.disableReferrals=true

This change disables the new feature addition in the upgraded JDK release and no longer sets the bit in the AS-REQ which advertises the Name Type. This effectively disables the feature additions for RFC-6806.

[#3]

Hi @GangWar ,

On the node this issue was occuring, somehow the default jdk folder name was java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64, so I assumed that it is 1.8.0.161, but after your suggestion I went into the node and issued "java -version" and to my surprise it was indeed openjdk version "1.8.0_252".

Following your suggestion has fixed the issue for me.

Thanks a lot!

Megh

[#4]

Please note, the instructions are also available in our public documentation under "Installing OpenJDK".

[#5]

HI @vidanimegh 

Ensure if you are able to do forward and reverse dns lookup., Iptables are off.
Perform CM agent hard restart. 

Whats the java version, There's this bug https://bugs.openjdk.java.net/browse/JDK-8215032 wherein Servers with Kerberos enabled stop functioning. That could be a possibility