Skip to main content
vidanimegh
vidanimegh
Gold ⭐️
February 3, 2021
Solved

Datanode can't connect to the namenode after enabling kerberos

Hello Everyone,

 

I've recently enabled Kerberos in my cluster and since then one of my datanodes is not able to connect to the namenode:

 

I see these entries in the namenode logs:

 

 

2021-02-03 12:06:15,699 INFO  ipc.Server (Server.java:saslProcess(1573)) - Auth successful for $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO  ipc.Server (Server.java:authorizeConnection(2235)) - Connection from <datanode-ip>:42328 for protocol org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol is unauthorized for user dn/<datanode-hostname>@<Realm-Name> (auth:PROXY) via $4E8100-MH1MCLUV65LO@<Realm-Name> (auth:KERBEROS)
2021-02-03 12:06:15,700 INFO  ipc.Server (Server.java:doRead(1006)) - Socket Reader #1 for port 8020: readAndProcess from client <datanode-ip> threw exception [org.apache.hadoop.security.authorize.AuthorizationException: User: $4E8100-MH1MCLUV65LO@<Realm-Name> is not allowed to impersonate dn/<datanode-hostname>@<Realm-Name>]

 

 

 

From the datanode logs:

 

 

2021-02-03 12:14:33,806 WARN  datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(225)) - Problem connecting to server: <namenode-hostname>/<namenode-ip>:8020

 

 

 

I've ensured that the hostnames are in lowercase and consistent in all the nodes.

 

Telnet is also happening from the datanode to namenode hostname on 8020 port.

 

Regenerating Keytabs and restarting everything also didn't work.

 

Any other areas to look into?

 

Thanks,

Megh

GangWar
GangWarAnswer
Platinum ⭐️⭐️
February 3, 2021

@vidanimegh what java version you are using? This seems you are hitting a well known issue with java.  Cloudera product releases using one of the following JDK versions:

 OpenJDK 1.8 u242 and later updates(example u252, u262 etc)
 OpenJDK 11.0.6 and later updates
 Oracle JDK 8u241 and later updates

are affected by this. Below are the instructions to mitigate this issue.  

 

Instructions:

On all impacted nodes, this issue has been solved pursuant to implementing the steps below based on the environment. Edit the java.security file located in the active JDK on each node. You can find this file at one of the following locations:

 <JAVA_HOME>/jre/lib/security/
 <JAVA_HOME>/conf/security 

Add or alter the sun.security.krb5.disableReferrals parameter so that it is set to true:

sun.security.krb5.disableReferrals=true

This change disables the new feature addition in the upgraded JDK release and no longer sets the bit in the AS-REQ which advertises the Name Type. This effectively disables the feature additions for RFC-6806.

Cheers! Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
View in thread

4 replies

GangWar
Platinum ⭐️⭐️
GangWarAnswer
Platinum ⭐️⭐️
February 3, 2021

@vidanimegh what java version you are using? This seems you are hitting a well known issue with java.  Cloudera product releases using one of the following JDK versions:

 OpenJDK 1.8 u242 and later updates(example u252, u262 etc)
 OpenJDK 11.0.6 and later updates
 Oracle JDK 8u241 and later updates

are affected by this. Below are the instructions to mitigate this issue.  

 

Instructions:

On all impacted nodes, this issue has been solved pursuant to implementing the steps below based on the environment. Edit the java.security file located in the active JDK on each node. You can find this file at one of the following locations:

 <JAVA_HOME>/jre/lib/security/
 <JAVA_HOME>/conf/security 

Add or alter the sun.security.krb5.disableReferrals parameter so that it is set to true:

sun.security.krb5.disableReferrals=true

This change disables the new feature addition in the upgraded JDK release and no longer sets the bit in the AS-REQ which advertises the Name Type. This effectively disables the feature additions for RFC-6806.

Translate
Cheers! Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
    vidanimegh
    vidanimeghAuthor
    Gold ⭐️
    February 3, 2021

    Hi @GangWar ,

     

    On the node this issue was occuring, somehow the default jdk folder name was java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64, so I assumed that it is 1.8.0.161, but after your suggestion I went into the node and issued "java -version" and to my surprise it was indeed openjdk version "1.8.0_252".

     

    Following your suggestion has fixed the issue for me.

     

    Thanks a lot!

     

    Megh

    Translate
    kingpin
    Gold ⭐️
    kingpin
    Gold ⭐️
    February 3, 2021

    HI @vidanimegh 

     

    Ensure if you are able to do forward and reverse dns lookup., Iptables are off.
    Perform CM agent hard restart. 

    Whats the java version, There's this bug https://bugs.openjdk.java.net/browse/JDK-8215032 wherein Servers with Kerberos enabled stop functioning. That could be a possibility

     

     

    Translate
      Terms & ConditionsAccessibility statement