question Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!! in Support Questions

In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

bala_dec2012 — Sun, 18 Feb 2018 21:34:18 GMT

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Shelton — Tue, 20 Feb 2018 05:23:48 GMT

There is an error with your auth_to_local entry for RULE:[2:$1......

In Ambari -> HDFS -> configs -> Advanced -> hadoop.security.auth_to_local

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Could you change it to the below note the RULE[2 ....line has not '@'

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
RULE:[2:$1@$0](^.*FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Please let me know if that helped

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

bala_dec2012 — Tue, 20 Feb 2018 06:07:28 GMT

Thanks @Geoffrey Shelton Okot

I have corrected hadoop.security.auth_to_local as you suggested. But unfortunately that didnt fix the problem. The AD user is still not able to execute hdfs commands.

Any clue is greatly appreciated!

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Shelton — Tue, 20 Feb 2018 06:47:38 GMT

@Bala K

Add the below line CAPATHS in the krb5.conf for cross-realm authentication, a database is needed to construct the authentication paths between the realms. From your previous posting add it below Note the (.) dot after the = sign on the second line

 [domain_realm] 
xxxx
xxxxxx
[capaths]
 FDA.EYFIDS.NET = {
  FDAQA.EYFIDS.NET = .
 }

On the KDC do you have a principal krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET ? you can check with

# kadmin.local 
list_princs

If not add as below

addprinc -e "aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal" krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET

In Ambari -> HDFS -> configs -> Advanced -> hadoop.security.auth_to_local Can you remove these 2 rules

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g 
RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Replace them with

RULE:[1:$1@$0](.*@FDA.EYFIDS.NET)s/@.*//

Regenerate all Hadoop service principals and their respective keytabs and restart all stale components

Try and revert

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

bala_dec2012 — Tue, 20 Feb 2018 08:12:36 GMT

Thanks Geoffrey Shelton Okot

I have corrected krb5.conf and hadoop.security.auth_to_local as you suggested.

listprincs also returned krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET.

Regenerated all keytabs, restarted all stale components.

But unforturnately, this didnt fix the issue. AD user is still not able to execute hdfs commands.

Any clue is greatly appreciated!

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Shelton — Tue, 20 Feb 2018 08:28:21 GMT

@Bala K
can the AD user successfully grab a ticket?

# kinit aduser@FDA.EYFIDS.NET

Then

# Klist

Can you share the above outputs and also the zip and share the following

/var/log/krb5default.log
/var/log/kadmind.log
/var/log/krb5kdc.log

Cheers

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Shelton — Tue, 20 Feb 2018 08:36:01 GMT

@Bala K

Also, ensure the clock is synchronized between the KDc and AD !!

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Shelton — Tue, 20 Feb 2018 18:56:49 GMT

@Bala K

The errors logged in krb5kdc.log

Feb 19 18:59:23 qa-hdp-mitkdc.fdaqa.eyfids.net krb5kdc[1064](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) : LOOKING_UP_SERVER: authtime 0, bigsql/qa-hdp-node5.fdaqa.eyfids.net@FDAQA.EYFIDS.NET for 

host/qa-hdp-node1.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node2.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node3.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node4.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node5.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database

FQDN is always required by Kerberos make sure the entries in /etc/hosts reflect that too. A workaround is to set the LDAP server name directly rather than relying on DNS.

Please do the necessary changes and revert.

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Shelton — Wed, 21 Feb 2018 04:29:37 GMT

@Bala K

Yes please add the x.x.x.x dev-addc.fda.eyfids.net dev-addc line in the /etc/hosts on the KDC server and to force TCP protocol between the KDC and AD Server and the following line in the krb5.conf:

[libdefaults]
udp_preference_limit = 1

Remember we want to eliminate the host not found error

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

bala_dec2012 — Fri, 09 Mar 2018 20:40:26 GMT

I resolved finally.

The netdom trust command contains password having special characters. So the netdom trust command was failing without showing any error and there by AD trust was failing without showing any error. I did setup trust in AD(Domains and trusts) then it worked.