Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Solved Go to solution
Highlighted

In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

New Contributor

I resolved finally.

The netdom trust command contains password having special characters. So the netdom trust command was failing without showing any error and there by AD trust was failing without showing any error. I did setup trust in AD(Domains and trusts) then it worked.

9 REPLIES 9

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Mentor

@Bala K

There is an error with your auth_to_local entry for RULE:[2:$1......

In Ambari -> HDFS -> configs -> Advanced -> hadoop.security.auth_to_local

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Could you change it to the below note the RULE[2 ....line has not '@'

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
RULE:[2:$1@$0](^.*FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Please let me know if that helped

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

New Contributor

Thanks @Geoffrey Shelton Okot

I have corrected hadoop.security.auth_to_local as you suggested. But unfortunately that didnt fix the problem. The AD user is still not able to execute hdfs commands.

Any clue is greatly appreciated!

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Mentor

@Bala K

Add the below line CAPATHS in the krb5.conf for cross-realm authentication, a database is needed to construct the authentication paths between the realms. From your previous posting add it below Note the (.) dot after the = sign on the second line

 [domain_realm] 
xxxx
xxxxxx
[capaths]
 FDA.EYFIDS.NET = {
  FDAQA.EYFIDS.NET = .
 } 

On the KDC do you have a principal krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET ? you can check with

# kadmin.local 
list_princs 

If not add as below

addprinc -e "aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal" krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET

In Ambari -> HDFS -> configs -> Advanced -> hadoop.security.auth_to_local Can you remove these 2 rules

RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g 
RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g 

Replace them with

RULE:[1:$1@$0](.*@FDA.EYFIDS.NET)s/@.*// 

Regenerate all Hadoop service principals and their respective keytabs and restart all stale components

Try and revert

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

New Contributor

Thanks Geoffrey Shelton Okot

I have corrected krb5.conf and hadoop.security.auth_to_local as you suggested.

listprincs also returned krbtgt/FDAQA.EYFIDS.NET@FDA.EYFIDS.NET.

Regenerated all keytabs, restarted all stale components.

But unforturnately, this didnt fix the issue. AD user is still not able to execute hdfs commands.

Any clue is greatly appreciated!

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Mentor

@Bala K
can the AD user successfully grab a ticket?

# kinit aduser@FDA.EYFIDS.NET

Then

# Klist 

Can you share the above outputs and also the zip and share the following

/var/log/krb5default.log
/var/log/kadmind.log
/var/log/krb5kdc.log

Cheers

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Mentor

@Bala K

Also, ensure the clock is synchronized between the KDc and AD !!

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Mentor

@Bala K

The errors logged in krb5kdc.log

Feb 19 18:59:23 qa-hdp-mitkdc.fdaqa.eyfids.net krb5kdc[1064](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) : LOOKING_UP_SERVER: authtime 0, bigsql/qa-hdp-node5.fdaqa.eyfids.net@FDAQA.EYFIDS.NET for 

host/qa-hdp-node1.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node2.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node3.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node4.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database 
host/qa-hdp-node5.fdaqa.eyfids.net@FDAQA.EYFIDS.NET, Server not found in Kerberos database

FQDN is always required by Kerberos make sure the entries in /etc/hosts reflect that too. A workaround is to set the LDAP server name directly rather than relying on DNS.

Please do the necessary changes and revert.

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

Mentor

@Bala K

Yes please add the x.x.x.x dev-addc.fda.eyfids.net dev-addc line in the /etc/hosts on the KDC server and to force TCP protocol between the KDC and AD Server and the following line in the krb5.conf:

[libdefaults]
udp_preference_limit = 1

Remember we want to eliminate the host not found error

Re: In the Kerberized cluster, AD users are not able to execute hdfs commands !!!

New Contributor

I resolved finally.

The netdom trust command contains password having special characters. So the netdom trust command was failing without showing any error and there by AD trust was failing without showing any error. I did setup trust in AD(Domains and trusts) then it worked.

Don't have an account?
Coming from Hortonworks? Activate your account here