AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

[#1]

Hello,

after enabling Kerberos, I created a user in AD and on host machine which is part of a group of superusers in CM:

than I set permissions in Ranger like so:

and after doing kinit and hdfs dfs -ls / I have an error:

WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

ls: DestHost:destPort FQDN:8020 , LocalHost:localPort FQDN/X.X.X.220:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

Could someone help please?

[#2]

@stale  Issue seems to be with java, Can you check the jdk version , try exporting the latest jdk version 

[#3]

@Scharan java version:

java-11-openjdk-11.0.15.0.9-2.el8_4.x86_64

Logs from debugging:

hdfs dfs -ls
                [UnixLoginModule]: succeeded importing info:
                        uid = 1012
                        gid = 491
                        supp gid = 491

Debug is  true storeKey false useTicketCache true useKeyTab false doNotPrompt true ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is true principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false

Refreshing Kerberos configuration
Acquire TGT from Cache
Principal is null
null credentials from Ticket Cache

                [Krb5LoginModule] authentication failed
Unable to obtain Principal Name for authentication

                [UnixLoginModule]: added UnixPrincipal,
                                UnixNumericUserPrincipal,
                                UnixNumericGroupPrincipal(s),
                         to Subject

But klist output shows principal:

klist

Ticket cache: KCM:1012
Default principal: hdfssu@DOMAIN.COM
 
Valid starting       Expires              Service principal
07/12/2022 13:20:29  07/12/2022 23:20:29  krbtgt/DOMAIN.COM@DOMAIN.COM
        renew until 07/19/2022 13:20:29

[#4]

@stale Can try running the below commands and share the output of debug logs

# export HADOOP_OPTS="-Dsun.security.krb5.debug=true"
# hdfs dfs -ls / 

[#5]

This error occurs because of hdfs is not able to locate keytab to run the command with permission.

Note: In the hadoop/hdfs file system webui you will be able enter to the path and see its permission and you will be able to open each file system paths by clicking in the webUI.
if try to modify/delete it will give error with your webui login permission (write)

If you are using kerberos authentication. follow below steps to run hdfs commands in any namenode. 

1. Copy the hdfs.keytab

       cd /run/cloudera-scm-agent/process

        ls -ltr to identify the latest dir for xxxxxxxxx-hdfs-NAMEBNODE
        cd xxxxxxxxx-hdfs-NAMEBNODE
        cp hdfs.keytab /  (any dir)
  2. Once copies run the kinit -kt command

        kinit -kt /hdfs.keytab hdfs/masterserver.domain.com@DOMAIN.COM
         klist
3. Run the hdfs command

      hdfs dfs -ls /