Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

The application won't work without a running HiveServer2 and Access denied erors after enabling LDAP

avatar
Rising Star

Cloudera Express 5.9.3

Hue 3.11

Sentry is enabled, admin groups are added to Admin Groups and admin users are added into Allowed Connecting Users in the Setry configuration.

 

SSSD is enabled on the servers, this means that users exist only in LDAP, but can log in into OS using their LDAP credentials. Admin users accounts exist in servers OS and match admin users accounts made in LDAP (and also match admin users accounts in Hue).

System account such as hue, hive, cloudera-scm etc aslo made in LDAP.

 

After enabling LDAP for Hue adn Hive I recieve errors

Potential misconfiguration detected. Fix and restart Hue.

Hive

The application won't work without a running HiveServer2.

When trying to enter Security - Hive tables an error Access denied to user occure.

 

Query Editors- Hive Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user) (code THRIFTTRANSPORT): TTransportException('Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user)',)

 

Hive log:

org.apache.thrift.transport.TTransportException: PLAIN auth failed: LDAP Authentication failed for user

 

 

I have tried all the advices, posted here, nothing helped.

2 ACCEPTED SOLUTIONS

avatar
Master Guru

@Paulina,

 

Long journey, so I hope this does the trick.

While not exposed in Cloudera Manager, you can identify an alternative userid attribute.  By default, for posix objecdts, the default is memberUid (which won't work for you unless memberUid contains the numeric id).

 

Try this:

 

- In Cloudera Manager, navigate to:

 

Clusters --> HDFS --> Configuration

 

- Search for:

 

Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml

 

- Add the following:

Name: hadoop.security.group.mapping.ldap.posix.attr.uid.name

Value: uid

 

- Save

- Restart the cluster (so the servers can detect the new settings)

 

BACKGROUND:

 

As you observed, by default HDFS assumes that the memberUid attribute will have a value of the uidNumber of the user account.  The configuration I mentioned above lets you adjust this so that rather than searching for "memberUid=1004" the search will contain "uid=maslova"

 

You can find more information here:

 

 

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml#hadoop.sec...

 

This is the description:
"The attribute of posixAccount to use when groups for membership. Mostly useful for schemas wherein groups have memberUids that use an attribute other than uidNumber."

 

As you can see, the configuration for hdfs meets the needs of your situation.

 

Cheers,

 

Ben

View solution in original post

avatar
Master Guru

@Paulina,

 

Sorry for the mistake... I meant that now with the suggested change, the resulting group lookup filter will be:

 

(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=maslova)))

 

This is because the hadoop.security.group.mapping.ldap.posix.attr.uid.name property tells ldapgroupmapping which user attribute to use to obtain the value for group lookup.  In your LDAP config that is "uid"

 

View solution in original post

21 REPLIES 21

avatar
Rising Star

What I found in the log of Sentry

 

DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [polymatica]

 

But why??????

 

The user maslova in not in group polymatica neither in LDAP nor in OS.

1.jpg2.jpg3.jpg4.jpg

 

All the mistakes disappear as I add group polymatica in Sentry admins, and in garnts for hive tables.

 

What is wrong?

avatar
Master Guru

@Paulina,

 

 

I agree that this is pretty odd, but we'll figure it out

 

We see that you have LDAP Groups Mapping configured for HDFS

I assume, then, that HDFS stuff appears to be working correctly.

 

Let's confirm by running:

 

# hdfs groups maslova

# hdfs groups hdfs

 

If Sentry is using the same lookup as HDFS, then we would assume the results would be the same and you would see that maslova is a member of ploymatica group.

However, if the "hdfs groups" command results differ, this indicates that Sentry is not using the same configuration for group lookups.

 

Let's start by getting those results.

 

avatar
Rising Star

Hello, bgooley!

 

hdfs groups maslova
maslova : polymatica

 

hdfs groups hdfs
hdfs : polymatica

 

Why does it happen? And how to fix it?

 

Thank you for your answers!

avatar
Master Guru

@Paulina,

 

What the result is telling you is that the the configuration you have for groups mapping in HDFS is returning that result.  Based on your LDAP and passwd output, it appears that your cluster is using a different means to derive group membership that you thought.

 

Let's check the following to confirm what clients and HDFS NameNodes are using for their groups mapping:

 

(1)

 

Client:

 

Run this on any host that is part of your cluster:

 

# grep -B1 -A2 "hadoop.security.group.mapping" /etc/hadoop/conf/core-site.xml

 

(2)

 

NameNode:

 

On your NameNode host, run the following:

 

# grep -B1 -A2 hadoop.security.group.mapping /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep "\-NAMENODE$"| tail -1`/core-site.xml

 

We can use the information there to help understand where hdfs is getting its mapping for your users.  The fact the "hdfs" user mapping to something other than "hdfs hadoop" for groups is disconcerting indeed.

 

I should also note that there is one other thing to consider.  If someone there has added values in the following property, that will create a static mapping that will override any other groups mapping:

 

hadoop.user.group.static.mapping.overrides

 

Check your server/client core-site.xml for that just to be case.  By default, it won't appear in core-site.xml

 

 

avatar
Rising Star

@bgooley

 

Hi

 

Host

 

  <property>

    <name>hadoop.security.group.mapping</name>

    <value>org.apache.hadoop.security.LdapGroupsMapping</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.url</name>

    <value>ldap://sspeapp01v.sec.oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.user</name>

    <value>cn=admin,dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.base</name>

    <value>dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

    <value>(objectClass=posixAccount)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

    <value>(objectClass=posixGroup)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

    <value>memberUid</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

    <value>cn</value>

  </property>

 

NameNode

 

<property>

    <name>hadoop.security.group.mapping</name>

    <value>org.apache.hadoop.security.LdapGroupsMapping</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.url</name>

    <value>ldap://sspeapp01v.sec.oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.user</name>

    <value>cn=admin,dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.password</name>

    <value>********</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.base</name>

    <value>dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

    <value>(objectClass=posixAccount)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

    <value>(objectClass=posixGroup)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

    <value>memberUid</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

    <value>cn</value>

  </property>

 

 

And there is no hadoop.user.group.static.mapping.overrides in core-site.xml

 

 

avatar
Master Guru

@Paulina,

 

It appears that your user search filter may need adjustment.

 

You have (objectclass=posixAccount) but are missing the part of the filter that accepts the uid.

You might try the following in Hadoop User Group Mapping LDAP User Search Filter:

 

(&(objectClass=posixAccount)(uid={0}))

 

I don't have time at the moment to go through the code more and figure out why you are seeing this exact behavior, but I think the above change is a good start.

 

avatar
Rising Star

@bgooley

 

Unfortunately that did not help

avatar
Master Guru

@Paulina,

 

The next step, then, is to find out what LDAP commands are being issued on the LDAP server.  Since you are using a non-AD server, your LDAP access logs should give us that information.  I would tail the ldap server access logs while doing the "hdfs groups hdfs" command for instance.

 

Once we see what ldap commands are run, we should have a better idea of why we are seeing the results we are seeing.

 

Also, you could use Wireshark or tcpdump to capture packets and view them in Wireshark.  Wireshark decodes LDAP packets so we can actually view the complete conversation between client and server.

 

 

avatar
Rising Star

@bgooley

 

 Hello!

 

The changes you have adviced in search filter  (&(objectClass=posixAccount)(uid={0})) had an effect!

 

Now the command hdfs groups hdfs returns hdfs : hdfs

 

hdfs groups maslova
maslova : maslova

 

Now the search bind returns the group, which is in gidNumber of the account.

 

LDAP log:

 

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixAccount)(uid=maslova))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (uid) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SEARCH RESULT tag=101 err=0 nentries=1 text=

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=1004)))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (gidNumber) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (memberUid) not indexed

 

 

HDFS configuration looks like:

 

Hadoop User Group Mapping LDAP Group Search Filter (objectClass=posixGroup)

Hadoop User Group Mapping LDAP Group Membership Attribute memberUid

Hadoop User Group Mapping LDAP Group Name Attribute cn

 

Something is wrong with these lines, I think

 

avatar
Rising Star

The reason of this problems was that HDFS is looking for uidNumber in memberUid and Hue is looking for uid in memberUid. As I added uidNumber in memberUid  for my user in addition to other memberUids everything started to work.

 

On the picture memberUid=1004 and memberUid=maslova is the same user

 

memberuid.jpg

 

Is there any other method to fix the problem, because if we leave membership like this we would have to add users in groups twice: uids and uidnumbers