@gerasimos,
I am still a bit unclear about your current deployment and where your users (not kerberos principals) exist now. It sounds as if you may be using OS files (/etc/passwd and /etc/group) for local users.
This page is a good start to help shape how you want to approach integration with AD...
https://www.cloudera.com/documentation/enterprise/5-16-x/topics/sg_auth_overview.html
Specifically, the AD integration section:
https://www.cloudera.com/documentation/enterprise/5-16-x/topics/sg_auth_overview.html
Also, at the bottom of the page is a matrix of what components support different authentication types.
Direct answers to your questions:
(1)
I can keep both existing user principals along with the AD users (on different realms). Is this right?
When talking about these topics, it is important to know two things: where your users' keberos accounts exist and where the hadoop service principals exist. If they are in the same realm, nothing special needs to be done. If your AD users have kerberos credentials in your AD realm, but your hadoop realm is different (and MIT Kerberos based) then you would need to configure cross-realm for your hadoop KDC to trust TGTs from the AD realm.
Yes, it is also perfectly reaonable to have our users authenticate via your MIT KDC and have users also authenticate, via LDAP with Active Directory (acting as an LDAP server).
(2)
For users controlled by AD, will I still need to create them in OS level?
Probably "YES", but it depends on what you want to do.
How your users' user/group mapping for hadoop is configured is up to you. Hadoop services need to be able to assess authorization via user/group mapping which defaults to org.apache.hadoop.security.ShellBasedUnixGroupsMapping. This means that shell commands such as "id" will be used to determine a user's group membership and thus their access to certain data. How the OS resolves the "id -Gn" command or others is up to your OS's configuration.
As mentioned in the documentation cited above, there are are several tools you can use in your cluster's nodes to integrate OS user/group with LDAP. SSSD is a free, open source one, for instance.
To summarize how authn/authz works for most components:
- User authenticates via Kerberos (Or LDAP if supported)
- the derived user id is then mapped to groups for authorization.
Front-end applications like Hue and Cloudera Manager are a bit different as they utilize internal service principals to access resources on behalf of the user who has authenticated to the UI. Hue, for instance, will connect as the "hue" service principal for the host it is on to proxy access for the user who has authenticated to Hue.
(3)
"A one-way, cross-realm trust must be set up from the local Kerberos realm to the central AD realm..."
Since I'm still a bit unsure what your endgame is, I hesitate to commit to anything here.
If you want all users to use their AD accounts but keep your CDH cluster's service principals in the MIT KDC, then, yes, you would need a one way trust where your CDH cluster's realm's KDC trusts the AD KDC's realm.
If you want your users to still kinit against your MIT KDC and your service principals are also there, no change is necessary.
I hope that helps a bit... if you can clear up where your users' principals will be and where your service principals will be, we can probably help confirm what basic work that entails.
LDAP auth and where LDAP accounts exist does not depend on Kerberos. The good part of AD is that accounts accessed for Kerberos and for LDAP exist as the same object.
If you have users on your MIT KDC for kerberos auth, but you want to introduce LDAP auth for services that supported it with AD as the LDAP server, you can do that... the only thing to keep in mind is that the user name (usually sAMAccountName) needs to be the same as their previous user name (whatever that was before LDAP) in order for authorization to work the same as it had before.
For example: if currently there is a user, "userone", who has access to certain HDFS directories and Sentry provides access to certain resources, after moving to LDAP auth, the username derived from LDAP authenitication must be "userone" as well.
