Unable to access hdfs directories after enabling kerberos!!

[#1]

Hi developers,
I have enabled kerberos on my cluster although cluster is green and running but i'm not able to access hdfs even after generating ticket.The error is same as the ticket is'nt generated. Probably authenticated server is not authenticating my ticket. Here is the error.

"19/05/16 08:53:07 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

ls: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "FQDN/X.X.X.X"; destination host is: "FQDN":PORT;"

Thanks.

[#2]

@Muhammad waqas

Can you share the below files?

• kadm5.acl
• kdc.conf
• krb5.conf

Switch to user hdfs from root account, please beware your output won't be exactly the same

# su - hdfs

Check if you have a valid ticket? You shouldn't have if the output isn't like below

$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_1013)

Destroy the Kerberos ticket

$ kdestroy

Get the principal attached to the hdfs keytab

$ klist -kt /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 10/11/2018 10:48:48 hdfs-host@HADOOP.COM
   1 10/11/2018 10:48:48 hdfs-host@HADOOP.COM
   1 10/11/2018 10:48:48 hdfs-host@HADOOP.COM
   1 10/11/2018 10:48:48 hdfs-host@HADOOP.COM
   1 10/11/2018 10:48:48 hdfs-host@HADOOP.COM

Grab a ticket by appending keytab + principal as below

$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab  hdfs-host@HADOOP.COM

Now you should have a validate kerberos ticket

$ klist
Ticket cache: FILE:/tmp/krb5cc_1013
Default principal: hdfs-host@HADOOP.COM
Valid starting       Expires              Service principal
05/16/2019 11:24:11  05/17/2019 11:24:11  krbtgt/HADOOP.COM@HADOOP.COM

Try to access hdfs

$ hdfs dfs -ls /

The above command should not error out.

[#3]

Operating system SuSE Linux sles 12 sp3

cloudera cdh 6.2

details of files mentioned below:

1: krb5.conf

[libdefaults]

default_realm = ABCDATA.ORG

dns_lookup_kdc = false

dns_lookup_realm = false

ticket_lifetime = 86400

renew_lifetime = 604800

forwardable = true

default_tgs_enctypes = aes256-cts-hmac-sha1-96

default_tkt_enctypes = aes256-cts-hmac-sha1-96

permitted_enctypes = aes256-cts-hmac-sha1-96

udp_preference_limit = 1

kdc_timeout = 3000

[realms]

ABCDATA.ORG = {

kdc = cloudera.abcdata.org

admin_server = cloudera.abcdata.org

}

2: kdc.conf

[kdcdefaults]

kdc_ports = 88

kdc_tcp_ports = 88

[realms]

ABCDATA.ORG = {

database_name = /var/lib/kerberos/krb5kdc/principal

admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab

acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl

dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict

key_stash_file = /var/lib/kerberos/krb5kdc/.k5.EXAMPLE.COM

kdc_ports = 88

max_life = 1d

max_renewable_life = 7d

}

[logging]

kdc = FILE:/var/log/krb5/krb5kdc.log

admin_server = FILE:/var/log/krb5/kadmind.log

3: kadm5.acl

###############################################################################

#Kerberos_principal permissions [target_principal] [restrictions]

###############################################################################

#

*/admin@ABCDATA.ORG *

[#4]

I've applied all the steps you've written above but unfortunately result is same. i dont know why my ticket is not being authenticatd!!!!

[#5]

@Muhammad waqas

I saw some discrepancy in the krb5.conf please copy and paste this one which I have updated with your entries

[libdefaults]
    default_realm = ABCDATA.ORG
    dns_lookup_realm = false
    dns_lookup_kdc = false
    ticket_lifetime = 24h
    forwardable = true
    udp_preference_limit = 1000000
    default_tgs_enctypes = aes256-cts-hmac-sha1-96
    default_tkt_enctypes = aes256-cts-hmac-sha1-96
    permitted_enctypes = aes256-cts-hmac-sha1-96
    kdc_timeout = 3000

[realms]
    ABCDATA.ORG = {
        kdc = cloudera.abcdata.org
        admin_server = cloudera.abcdata.org
        default_domain = ABCDATA.ORG
    }

[domain_realm]
    .abcdata.org = ABCDATA.ORG
     abcdata.org = ABCDATA.ORG

[logging]
    kdc = FILE:/var/log/krb5kdc.log
    admin_server = FILE:/var/log/kadmin.log
    default = FILE:/var/log/krb5lib.log

Problem

[TOKEN, KERBEROS]; Host Details: local host is: "FQDN/X.X.X.X"; destination host is: "FQDN":PORT;"

The above shows your hostname is not configured

# kadmin.local 
Authenticating as principal root/admin@ABCDATA.ORG with password. 
kadmin.local:  listprincs

Sample output on Hortonworks

nm/cloudera.abcdata.org@ABCDATA.ORG 
nn/cloudera.abcdata.org@ABCDATA.ORG 
oozie/cloudera.abcdata.org@ABCDATA.ORG 
rangeradmin/cloudera.abcdata.org@ABCDATA.ORG 
rangerlookup/cloudera.abcdata.org@ABCDATA.ORG 
rangertagsync/cloudera.abcdata.org@ABCDATA.ORG 
rangerusersync/cloudera.abcdata.org@ABCDATA.ORG 
rm/cloudera.abcdata.org@ABCDATA.ORG

Can you share the output of

$ hostname -f

Does it match the entries in /etc/hosts? the format should be

IP:FQDN:ALIAS

After the validation and correction please regenerate the keytabs using Cloudera Manager Admin Console

HTH