Skip to main content
gerasimos
gerasimos
Gold ⭐️
January 22, 2019
Solved

Clarifications on LDAP integration

Hello,

 

I have a  kerberized 5.14 CDH cluster and I want to integrate with LDAP for user authentication (not service). I suppose this is what described in "Local MIT KDC with Active Directory Integration".

 

I have the following questions about this change, if I have understood the process right:

 

1. Users now will be defined in AD and not kerberos. This means that current kerberos keytab files will no longer be valid. Right?

 

2. In case of CLI pipelines, where we first have to do a kinit for the principal, how the authentication will be after LDAP? 

 

3. Which services should be configured to work with LDAP?

- Cloudera Manager

- Hue

- Hive

- Impala

- ... ?

 

4. Will groups and Sentry permissions require to be re-configured after enabling LDAP? 

 

Thank you,

gerasimos

bgooley
bgooleyAnswer
Gold ⭐️
January 23, 2019

@gerasimos,

 

I am still a bit unclear about your current deployment and where your users (not kerberos principals) exist now.  It sounds as if you may be using OS files (/etc/passwd and /etc/group) for local users.

 

This page is a good start to help shape how you want to approach integration with AD... 

https://www.cloudera.com/documentation/enterprise/5-16-x/topics/sg_auth_overview.html

 

Specifically, the AD integration section:

 

https://www.cloudera.com/documentation/enterprise/5-16-x/topics/sg_auth_overview.html

 

Also, at the bottom of the page is a matrix of what components support different authentication types.

 

Direct answers to your questions:

 

(1)

 

I can keep both existing user principals along with the AD users (on different realms). Is this right?

 

When talking about these topics, it is important to know two things:  where your users' keberos accounts exist and where the hadoop service principals exist.  If they are in the same realm, nothing special needs to be done.  If your AD users have kerberos credentials in your AD realm, but your hadoop realm is different (and MIT Kerberos based) then you would need to configure cross-realm for your hadoop KDC to trust TGTs from the AD realm.

 

Yes, it is also perfectly reaonable to have our users authenticate via your MIT KDC and have users also authenticate, via LDAP with Active Directory (acting as an LDAP server).

 

(2)

 

For users controlled by AD, will I still need to create them in OS level?

 

Probably "YES", but it depends on what you want to do.

 

How your users' user/group mapping for hadoop is configured is up to you.  Hadoop services need to be able to assess authorization via user/group mapping which defaults to org.apache.hadoop.security.ShellBasedUnixGroupsMapping.  This means that shell commands such as "id" will be used to determine a user's group membership and thus their access to certain data.  How the OS resolves the "id -Gn" command or others is up to your OS's configuration. 

 

As mentioned in the documentation cited above, there are are several tools you can use in your cluster's nodes to integrate OS user/group with LDAP.  SSSD is a free, open source one, for instance.

 

To summarize how authn/authz works for most components:

 

- User authenticates via Kerberos (Or LDAP if supported)

- the derived user id is then mapped to groups for authorization.

 

Front-end applications like Hue and Cloudera Manager are a bit different as they utilize internal service principals to access resources on behalf of the user who has authenticated to the UI.  Hue, for instance, will connect as the "hue" service principal for the host it is on to proxy access for the user who has authenticated to Hue.

 

(3)

 

"A one-way, cross-realm trust must be set up from the local Kerberos realm to the central AD realm..."

 

Since I'm still a bit unsure what your endgame is, I hesitate to commit to anything here.

 

If you want all users to use their AD accounts but keep your CDH cluster's service principals in the MIT KDC, then, yes, you would need a one way trust where your CDH cluster's realm's KDC trusts the AD KDC's realm.

 

If you want your users to still kinit against your MIT KDC and your service principals are also there, no change is necessary.

 

 

I hope that helps a bit... if you can clear up where your users' principals will be and where your service principals will be, we can probably help confirm what basic work that entails.

LDAP auth and where LDAP accounts exist does not depend on Kerberos.  The good part of AD is that accounts accessed for Kerberos and for LDAP exist as the same object.

 

If you have users on your MIT KDC for kerberos auth, but you want to introduce LDAP auth for services that supported it with AD as the LDAP server, you can do that... the only thing to keep in mind is that the user name (usually sAMAccountName) needs to be the same as their previous user name (whatever that was before LDAP) in order for authorization to work the same as it had before.

 

For example:  if currently there is a user, "userone", who has access to certain HDFS directories and Sentry provides access to certain resources, after moving to LDAP auth, the username derived from LDAP authenitication must be "userone" as well.

 

View in thread

10 replies

bgooley
Gold ⭐️
bgooley
Gold ⭐️
January 22, 2019

 Hi @gerasimos,

 

First, I think you may be confusing LDAP and Kerberos here so we need some clarification on what you are actually trying to achieve.  For authentication.  Kerberos and LDAP authentication are used for different things so it is important to understand how each is utilized.

 

For authentication in CDH, Kerberos is required for core hadoop (HDFS, YARN) and clients of those.  Servers that are user-facing usually have an option of authenticating via LDAP or Kerberos.  Internal is all Kerberos, external can be a mixture depending on what you would like and what the servers support.

 

Bottom line, for hadoop security, you need Kerberos, but some things allow for LDAP too.

 

Also, LDAP can be utilized for user-group mapping.  I don't think that is what you are discussing.

 

Another thing to note is that "Kerberos" is not a server, it is a protocol.  Any Kerberos v5 server is fine, but CDH requires MIT Kerberos client libraries for communication in non-Java Kerberos operations.

 

To your questions:

 

(1)

 

Users are created and stored where you want them.  Active Directory hosts are also Kerberos servers in Windows systems so you can Use MIT Kerberos's KDC or Windows Active Directory as your Kerberos server.  Active Directory also supports LDAP protocol requests.  It is up to your deployment where your end users exist.

 

We'd need clarification about what configuration changes you are proposing to answer this better.

 

(2)

 

Need some clarification about what sort of "pipelines" you mean.  What services are they connecting to?

 

(3)

 

No service "should" be configured with LDAP meaning there is no functional gain to authentication with LDAP vs Kerberos.  Each will serve the purpose of authenticating.

 

The following can support LDAP authentication:

 

- Cloudera Manager

- Cloudera Navigator Metadata Server

- Hue

- Hive

- Impala

- Solr

- Sentry (in unsupported 'test' mode only - only Kerberos is supported)

 

(4)

No, no Sentry changes should be necessary as long as your groups and users are the same.

 

Permissions are group-based so that has nothing to do the means of authentication.  Sentry provides authorization protection.

 

To summarize:

- authentication by end users can be achieved by the methods supported by the various servers (roles).  Some support LDAP, Kerberos, SAML, etc.

- Once a user is authenticated, then authorization is evaluated depending on the role being accessed.

 

Hope that helps get started.

 

Translate
gerasimos
gerasimosAuthor
Gold ⭐️
January 23, 2019

Hello @bgooley,

 

Thank you for the detailed explanation. To clarify myself, when I said "kerberos" I meant the MIT KDC implementation, and yes I do not know much about LDAP and AD.

 

My organization has an Microsoft AD. It also has a CDH that uses MIT Kerberos for hadoop user and services authentication. CM and Hue have their own users.

 

The task is to review what needs to be done in order to have users declared in AD to use the cluster, e.g. for submiting Spark jobs, executing Impala queries, use CM, Hue etc.

 

As far as I have undestood, I can keep both existing user principals along with the AD users (on different realms). Is this right?

 

For users controlled by AD, will I still need to create them in OS level? If no, how HDFS user and group permissions are affected?

 

After your reply, I read again the link above, and I think that the key in this task is to undestand this:

"A one-way, cross-realm trust must be set up from the local Kerberos realm to the central AD realm containing the user principals that require access to the CDH cluster".

 

Thank you again for your effort.

Translate
bgooley
Gold ⭐️
bgooleyAnswer
Gold ⭐️
January 23, 2019

@gerasimos,

 

I am still a bit unclear about your current deployment and where your users (not kerberos principals) exist now.  It sounds as if you may be using OS files (/etc/passwd and /etc/group) for local users.

 

This page is a good start to help shape how you want to approach integration with AD... 

https://www.cloudera.com/documentation/enterprise/5-16-x/topics/sg_auth_overview.html

 

Specifically, the AD integration section:

 

https://www.cloudera.com/documentation/enterprise/5-16-x/topics/sg_auth_overview.html

 

Also, at the bottom of the page is a matrix of what components support different authentication types.

 

Direct answers to your questions:

 

(1)

 

I can keep both existing user principals along with the AD users (on different realms). Is this right?

 

When talking about these topics, it is important to know two things:  where your users' keberos accounts exist and where the hadoop service principals exist.  If they are in the same realm, nothing special needs to be done.  If your AD users have kerberos credentials in your AD realm, but your hadoop realm is different (and MIT Kerberos based) then you would need to configure cross-realm for your hadoop KDC to trust TGTs from the AD realm.

 

Yes, it is also perfectly reaonable to have our users authenticate via your MIT KDC and have users also authenticate, via LDAP with Active Directory (acting as an LDAP server).

 

(2)

 

For users controlled by AD, will I still need to create them in OS level?

 

Probably "YES", but it depends on what you want to do.

 

How your users' user/group mapping for hadoop is configured is up to you.  Hadoop services need to be able to assess authorization via user/group mapping which defaults to org.apache.hadoop.security.ShellBasedUnixGroupsMapping.  This means that shell commands such as "id" will be used to determine a user's group membership and thus their access to certain data.  How the OS resolves the "id -Gn" command or others is up to your OS's configuration. 

 

As mentioned in the documentation cited above, there are are several tools you can use in your cluster's nodes to integrate OS user/group with LDAP.  SSSD is a free, open source one, for instance.

 

To summarize how authn/authz works for most components:

 

- User authenticates via Kerberos (Or LDAP if supported)

- the derived user id is then mapped to groups for authorization.

 

Front-end applications like Hue and Cloudera Manager are a bit different as they utilize internal service principals to access resources on behalf of the user who has authenticated to the UI.  Hue, for instance, will connect as the "hue" service principal for the host it is on to proxy access for the user who has authenticated to Hue.

 

(3)

 

"A one-way, cross-realm trust must be set up from the local Kerberos realm to the central AD realm..."

 

Since I'm still a bit unsure what your endgame is, I hesitate to commit to anything here.

 

If you want all users to use their AD accounts but keep your CDH cluster's service principals in the MIT KDC, then, yes, you would need a one way trust where your CDH cluster's realm's KDC trusts the AD KDC's realm.

 

If you want your users to still kinit against your MIT KDC and your service principals are also there, no change is necessary.

 

 

I hope that helps a bit... if you can clear up where your users' principals will be and where your service principals will be, we can probably help confirm what basic work that entails.

LDAP auth and where LDAP accounts exist does not depend on Kerberos.  The good part of AD is that accounts accessed for Kerberos and for LDAP exist as the same object.

 

If you have users on your MIT KDC for kerberos auth, but you want to introduce LDAP auth for services that supported it with AD as the LDAP server, you can do that... the only thing to keep in mind is that the user name (usually sAMAccountName) needs to be the same as their previous user name (whatever that was before LDAP) in order for authorization to work the same as it had before.

 

For example:  if currently there is a user, "userone", who has access to certain HDFS directories and Sentry provides access to certain resources, after moving to LDAP auth, the username derived from LDAP authenitication must be "userone" as well.

 

Translate
    Terms & ConditionsAccessibility statement