Support Questions
Find answers, ask questions, and share your expertise

Datanode can't connect to the namenode after enabling kerberos

Solved Go to solution

Datanode can't connect to the namenode after enabling kerberos

Expert Contributor

Hello Everyone,

 

I've recently enabled Kerberos in my cluster and since then one of my datanodes is not able to connect to the namenode:

 

I see these entries in the namenode logs:

 

 

2021-02-03 12:06:15,699 INFO  ipc.Server (Server.java:saslProcess(1573)) - Auth successful for $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO  ipc.Server (Server.java:authorizeConnection(2235)) - Connection from <datanode-ip>:42328 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/<datanode-hostname>@<Realm-Name> (auth:PROXY) via $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO  ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client <datanode-ip> threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $4E8100-MH1MCLUV65LO@<Realm-Name> is not allowed to impersonate dn/<datanode-hostname>@<Realm-Name>]

 

 

 

From the datanode logs:

 

 

2021-02-03 12:14:33,806 WARN  datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(225)) - Problem connecting to server: <namenode-hostname>/<namenode-ip>:8020

 

 

 

I've ensured that the hostnames are in lowercase and consistent in all the nodes.

 

Telnet is also happening from the datanode to namenode hostname on 8020 port.

 

Regenerating Keytabs and restarting everything also didn't work.

 

Any other areas to look into?

 

Thanks,

Megh

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Datanode can't connect to the namenode after enabling kerberos

Master Collaborator

@vidanimegh what java version you are using? This seems you are hitting a well known issue with java.  Cloudera product releases using one of the following JDK versions:

 OpenJDK 1.8 u242 and later updates(example u252, u262 etc)
 OpenJDK 11.0.6 and later updates
 Oracle JDK 8u241 and later updates

are affected by this. Below are the instructions to mitigate this issue.  

 

Instructions:

On all impacted nodes, this issue has been solved pursuant to implementing the steps below based on the environment. Edit the java.security file located in the active JDK on each node. You can find this file at one of the following locations:

 <JAVA_HOME>/jre/lib/security/
 <JAVA_HOME>/conf/security 

Add or alter the sun.security.krb5.disableReferrals parameter so that it is set to true:

sun.security.krb5.disableReferrals=true

This change disables the new feature addition in the upgraded JDK release and no longer sets the bit in the AS-REQ which advertises the Name Type. This effectively disables the feature additions for RFC-6806.


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

4 REPLIES 4

Re: Datanode can't connect to the namenode after enabling kerberos

Master Collaborator

@vidanimegh what java version you are using? This seems you are hitting a well known issue with java.  Cloudera product releases using one of the following JDK versions:

 OpenJDK 1.8 u242 and later updates(example u252, u262 etc)
 OpenJDK 11.0.6 and later updates
 Oracle JDK 8u241 and later updates

are affected by this. Below are the instructions to mitigate this issue.  

 

Instructions:

On all impacted nodes, this issue has been solved pursuant to implementing the steps below based on the environment. Edit the java.security file located in the active JDK on each node. You can find this file at one of the following locations:

 <JAVA_HOME>/jre/lib/security/
 <JAVA_HOME>/conf/security 

Add or alter the sun.security.krb5.disableReferrals parameter so that it is set to true:

sun.security.krb5.disableReferrals=true

This change disables the new feature addition in the upgraded JDK release and no longer sets the bit in the AS-REQ which advertises the Name Type. This effectively disables the feature additions for RFC-6806.


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

Re: Datanode can't connect to the namenode after enabling kerberos

Expert Contributor

Hi @GangWar ,

 

On the node this issue was occuring, somehow the default jdk folder name was java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64, so I assumed that it is 1.8.0.161, but after your suggestion I went into the node and issued "java -version" and to my surprise it was indeed openjdk version "1.8.0_252".

 

Following your suggestion has fixed the issue for me.

 

Thanks a lot!

 

Megh

Re: Datanode can't connect to the namenode after enabling kerberos

Moderator

Please note, the instructions are also available in our public documentation under "Installing OpenJDK".


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Re: Datanode can't connect to the namenode after enabling kerberos

Contributor

HI @vidanimegh 

 

Ensure if you are able to do forward and reverse dns lookup., Iptables are off.
Perform CM agent hard restart. 

Whats the java version, There's this bug https://bugs.openjdk.java.net/browse/JDK-8215032 wherein Servers with Kerberos enabled stop functioning. That could be a possibility