LDAP user is treated as 'admin' by HDFS commands after kerberization. But treated as correct LDAP user if i disable kerberos. Plz help!!!

[#1]

Please provide an example rule that auto handles kerberos principals of all LDAP users

Rule in Ambari -> HDFS -> Configs -> "hadoop.security.auth_to_local"

wbinfo -u output:

user1

user2

....

[#2]

@Bala K

I think more information is needed. LDAP users are not always directly related to Kerberos identities. However if you are using an Active Director or IPA server they will appear to be.

By default, Ambari should set up a default rule to translate the user principals to local user names. If there are multiple realms, the additional realms may be declared to Ambari, for which each will have a rule created. This property can be found on the Admin/Kerberos page.

If you wish to create custom rules for the users, see my article on writing auth-to-local-rules - https://community.hortonworks.com/articles/14463/auth-to-local-rules-syntax.html

[#3]

Thanks Robert for quick reply.

Let me elaborate the problem description:

Our HDP2.6 cluster is kerberized and integrated with LDAP. Now, We want all LDAP users to be able to login to our cluster and execute HDFS commands.

When any AD user(Ex: bspchaseuser) logs into putty and executes any HDFS command, that AD user is treated as 'admin' by HDFS instead of his actual user name. Please see below.

One thing i noticed is,

If i change hadoop.security.authentication value from 'kerberos' to 'simple' in Ambari -> HDFS -> Configs -> Advanced core-site

Then HDFS commands treat the logged in user correctly as AD user but not as 'admin'.

Please help me, what is the resolution ?

More info:

Our AD Domain is in the format XYZ.ABC.NET

Our KDC Domain and HDP cluster Domain are in the format XYZDEV.ABC.NET

Ambari -> Admin -> Kerberos -> Realm is also XYZDEV.ABC.NET

The extra principal Ambari created apart from service principals is, RULE:[1:$1@$0](.*@XYZDEV.ABC.NET)s/@.*//

Do i need to manually create principals in KDC server for each of the AD user ?

//There are 100's of users, practically difficult. I created for 1 user but didnt work.

Do i need to manually declare any new RULE under Ambari -> HDFS ->Configs -> hadoop.security.auth_to_local ?

//I tried different ways in writing the rule, but didnt work. Can you please help with the exact rule to be kept? You have all details.

Thanks

[#4]

Since you have the cluster set up for Kerberos, you should leave that hadoop.security.authentication property set to kerberos. By setting it to simple, it Hadoop will use the acting users local username rather than the authenticated Kerberos principal. This is less secure and can introduce issues since some services will use the local username and others will use the Kerberos identity.

What are you using to integrate the AD with the Linux hosts? Maybe the configuration of this is causing the wrong Kerberos principal to be used when logging in to the hosts. After logging in, do a klist to see what Kerberos identity you have acquired. If this is not the expected principal, you should review that integration piece. So people use SSSD for this, but there are other tools out there. I am not familiar with any, as I am not a system administrator and have not had the time to do any research on them.

[#5]

Thanks Robert for your quick reply.

I kept the hadoop.security.authentication property to 'Kerberos' now. We used Winbind to integrate AD with Linux hosts. That is working fine. Im able to see the users and groups with 'wbinfo -u' and 'wbinfo -g'. But when i log in as AD user and execute "HDFS groups", then its not showing the AD group of that logged in AD user.

After logging in as AD user, when i do klist, its returning

Default principal: admin/admin@XYZDEV.ABC.NET

1)Should i expect the return value to be bspchaseuser/bspchase@XYZ.ABC.NET ?

//bspchaseuser is the AD user and bspchase is the AD user's AD group. XYZ.ABC.NET is the AD domain.

2)Do i need to manually create principals in KDC server for each of the AD user ?

//There are 100's of users, practically difficult. I created for 1 user but didnt work.

3)Do i need to manually declare any new RULE under Ambari -> HDFS ->Configs -> hadoop.security.auth_to_local ?

//I tried different ways in writing the rule, but didnt work. Can you please help with the exact rule to be kept? You have all details.

I spent a week trying to fix this. Can you please guide me to the right channel.

Thanks.

[#6]

Winbind is out of my league, but I would think that when you log in to the linux host as user bspchaseuser, winbind should automatically kinit for you and your principal should be bspchaseuser@XYZ.ABC.NET not admin/admin@XYZDEV.ABC.NET. So it seems like something is not correct. Or maybe winbind does not handle Kerberos and you are required to kinit separately. Since the realms are different, it seems like you, at one point, kinit-ed as admin/admin@XYZDEV.ABC.NET for the local KDC and that identity is hanging around each time you log in due to the stored Kerberos credential cache. Maybe you need to clear out the cache using kdestroy and then winbind will re-establish the cache using the expected identity.

Regarding the KDC, if you are using AD, you really do not need an additional KDC... however I typically recommend using an MIT KDC for the cluster's service principals. You will need to create a trust relationship between the local KDC and the AD to properly authenticate the enterprise users in the XYZ.ABC.NET realm and tell Ambari about the additional realm (XYZ.ABC.NET) so that it knows how to properly create the auth-to-local rules. It does nothing else with that information. So you will also need to edit the krb5.conf template so the realm can be found by the Kerberos infrastructure.

You do not need to perform actions for your #2 and #3 items above if Winbind and Ambari are set up correctly.

[#7]

Many thanks Robert.

You are right. Unknowngly, I did kinit admin/admin@XYZDEV.ABC.NET earlier. That caused the problem. But now i did kinit bspchaseuser@XYZDEV.ABC.NET and i'm able to execute the HDFS commands succesfully. BUT ONLY if i add the principal for bspchaseuser manually in KDC server.

Problem description:

Not sure, where im making mistake. But the AD users can login and execute 'HDFS groups' command only if i manually add a principal for each of them with their username as shown below.

More info:

- Cluster is Kerberized with MIT KDC and integrated with AD using Winbind.

- Everything is working, except i need to manually add principal for each AD user in MIT KDC server.

krb5.conf in all nodes is as below:

[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = FDADEV.EYFIDS.NET
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
default_ccache_name = /tmp/krb5cc_%{uid}
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log

[realms]
FDADEV.EYFIDS.NET = {
admin_server = dev-hdp-mitkdc.fdadev.eyfids.net
kdc = dev-hdp-mitkdc.fdadev.eyfids.net

default_domain = fdadev.eyfids.net

}

FDA.EYFIDS.NET = {
kdc = dev-addc.fda.eyfids.net
admin_server = dev-addc.fda.eyfids.net
default_domain = fda.eyfids.net
}

As per the link https://community.hortonworks.com/articles/59635/one-way-trust-mit-kdc-to-active-directory.html

AD trust with MIT KDC is established and tried by adding below rules also

#RULE:[1:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g
#RULE:[2:$1@$0](^.*@FDA\.EYFIDS\.NET$)s/^(.*)@FDA\.EYFIDS\.NET$/$1/g

Can you please let me know how to get principals of AD users added automatically instead of manually adding the principal for each of the AD user in MIT KDC server.

[#8]

@Bala K

It seems like we are getting closer. So I think we have some confusion on what realm the enterprise users need to be. You posted that bspchaseuser@XYZDEV.ABC.NET was the user you used, but needed to create a principal in the cluster's MIT KDC. However, if you wanted to use the enterprise user identity, the principal should have been bspchaseuser@XYZ.ABC.NET. Or looking at your krb5.conf file, bspchaseuser@FDA.EYFIDS.NET. Since Winbind appears to not automatically kinit for you, you need to issue the following kinit command after logging in:

kinit bspchaseuser@FDA.EYFIDS.NET

If the trust is set up properly, you should be able do successfully issue an HDFS command

hdfs dfs -ls /

If something it wrong you will get some Kerberos-related or GSS-API error. \

[#9 · Accepted solution]

Many Thanks Robert. You solved the problem.

kinit bspchaseuser@XYZ.ABC.NET worked successfully. I have been wrongly trying bspchaseuser@XYZDEV.ABC.NET

By the way, the TGT ticket expires everyday. Does every AD user need to kinit everyday to be able to perform on the cluster ? Also, do i need to do kinit on the hadoop services (hdfs, hive, hbase, etc.) every day to renew the expired ticket ?

Thanks.

[#10]

Usually by default the ticket expires 24 hours and cache expires 7 days. Depends on your directory services policies. Within 7 days you can do kinit -R for users.

klist will show ticket and cache expiry time. Or you can use keytabs to automate ticket renewal.

You don't have to kinit for hadoop services (ever), ticket renewal is managed automatically.