In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

[#1]

@Bala K

There is an error with your auth_to_local entry for RULE:[2:$1......

In Ambari -> HDFS -> configs -> Advanced -> hadoop.security.auth_to_local

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Could you change it to the below note the RULE[2 ....line has not '@'

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
RULE:[2:$1@$0](^.*FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Please let me know if that helped

[#2]

Thanks @Geoffrey Shelton Okot

I have corrected hadoop.security.auth_to_local as you suggested. But unfortunately that didnt fix the problem. The AD user is still not able to execute hdfs commands.

Any clue is greatly appreciated!

[#3]

@Bala K

Add the below line CAPATHS in the krb5.conf for cross-realm authentication, a database is needed to construct the authentication paths between the realms. From your previous posting add it below Note the (.) dot after the = sign on the second line

 [domain_realm] 
xxxx
xxxxxx
[capaths]
 FDA.EYFIDS.NET = {
  FDAQA.EYFIDS.NET = .
 } 

On the KDC do you have a principal krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET ? you can check with

# kadmin.local 
list_princs 

If not add as below

addprinc -e "aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal" krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET

In Ambari -> HDFS -> configs -> Advanced -> hadoop.security.auth_to_local Can you remove these 2 rules

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g 
RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g 

Replace them with

RULE:[1:$1@$0](.*@FDA.EYFIDS.NET)s/@.*// 

Regenerate all Hadoop service principals and their respective keytabs and restart all stale components

Try and revert

[#4]

Thanks Geoffrey Shelton Okot

I have corrected krb5.conf and hadoop.security.auth_to_local as you suggested.

listprincs also returned krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET.

Regenerated all keytabs, restarted all stale components.

But unforturnately, this didnt fix the issue. AD user is still not able to execute hdfs commands.

Any clue is greatly appreciated!

[#5]

@Bala K
can the AD user successfully grab a ticket?

# kinit aduser@FDA.EYFIDS.NET

Then

# Klist 

Can you share the above outputs and also the zip and share the following

/var/log/krb5default.log
/var/log/kadmind.log
/var/log/krb5kdc.log

Cheers

[#6]

@Bala K

Also, ensure the clock is synchronized between the KDc and AD !!

[#7]

@Bala K

The errors logged in krb5kdc.log

Feb 19 18:59:23 qa-hdp-mitkdc.fdaqa.eyfids.net krb5kdc[1064](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) : LOOKING_UP_SERVER: authtime 0, bigsql/qa-hdp-node5.fdaqa.eyfids.net@FDAQA.EYFIDS.NET for 

host/qa-hdp-node1.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node2.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node3.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node4.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node5.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database

FQDN is always required by Kerberos make sure the entries in /etc/hosts reflect that too. A workaround is to set the LDAP server name directly rather than relying on DNS.

Please do the necessary changes and revert.

[#8]

@Bala K

Yes please add the x.x.x.x dev-addc.fda.eyfids.net dev-addc line in the /etc/hosts on the KDC server and to force TCP protocol between the KDC and AD Server and the following line in the krb5.conf:

[libdefaults]
udp_preference_limit = 1

Remember we want to eliminate the host not found error

[#9 · Accepted solution]

I resolved finally.

The netdom trust command contains password having special characters. So the netdom trust command was failing without showing any error and there by AD trust was failing without showing any error. I did setup trust in AD(Domains and trusts) then it worked.