The application won't work without a running HiveServer2 and Access denied erors after enabling LDAP

[#1]

Cloudera Express 5.9.3

Hue 3.11

Sentry is enabled, admin groups are added to Admin Groups and admin users are added into Allowed Connecting Users in the Setry configuration.

SSSD is enabled on the servers, this means that users exist only in LDAP, but can log in into OS using their LDAP credentials. Admin users accounts exist in servers OS and match admin users accounts made in LDAP (and also match admin users accounts in Hue).

System account such as hue, hive, cloudera-scm etc aslo made in LDAP.

After enabling LDAP for Hue adn Hive I recieve errors

Potential misconfiguration detected. Fix and restart Hue.

Hive	The application won't work without a running HiveServer2.

When trying to enter Security - Hive tables an error Access denied to user occure.

Query Editors- Hive Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user) (code THRIFTTRANSPORT): TTransportException('Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user)',)

Hive log:

org.apache.thrift.transport.TTransportException: PLAIN auth failed: LDAP Authentication failed for user

I have tried all the advices, posted here, nothing helped.

[#2 · Accepted solution]

@Paulina,

Long journey, so I hope this does the trick.

While not exposed in Cloudera Manager, you can identify an alternative userid attribute.  By default, for posix objecdts, the default is memberUid (which won't work for you unless memberUid contains the numeric id).

Try this:

- In Cloudera Manager, navigate to:

Clusters --> HDFS --> Configuration

- Search for:

Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml

- Add the following:

Name: hadoop.security.group.mapping.ldap.posix.attr.uid.name

Value: uid

- Save

- Restart the cluster (so the servers can detect the new settings)

BACKGROUND:

As you observed, by default HDFS assumes that the memberUid attribute will have a value of the uidNumber of the user account.  The configuration I mentioned above lets you adjust this so that rather than searching for "memberUid=1004" the search will contain "uid=maslova"

You can find more information here:

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml#hadoop.sec...

This is the description:
"The attribute of posixAccount to use when groups for membership. Mostly useful for schemas wherein groups have memberUids that use an attribute other than uidNumber."

As you can see, the configuration for hdfs meets the needs of your situation.

Cheers,

Ben

[#3]

What I found in the log of Sentry

DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [polymatica]

But why??????

The user maslova in not in group polymatica neither in LDAP nor in OS.

All the mistakes disappear as I add group polymatica in Sentry admins, and in garnts for hive tables.

What is wrong?

[#4]

@Paulina,

I agree that this is pretty odd, but we'll figure it out

We see that you have LDAP Groups Mapping configured for HDFS

I assume, then, that HDFS stuff appears to be working correctly.

Let's confirm by running:

# hdfs groups maslova

# hdfs groups hdfs

If Sentry is using the same lookup as HDFS, then we would assume the results would be the same and you would see that maslova is a member of ploymatica group.

However, if the "hdfs groups" command results differ, this indicates that Sentry is not using the same configuration for group lookups.

Let's start by getting those results.

[#5]

Hello, bgooley!

hdfs groups maslova
maslova : polymatica

hdfs groups hdfs
hdfs : polymatica

Why does it happen? And how to fix it?

Thank you for your answers!

[#6]

@Paulina,

What the result is telling you is that the the configuration you have for groups mapping in HDFS is returning that result.  Based on your LDAP and passwd output, it appears that your cluster is using a different means to derive group membership that you thought.

Let's check the following to confirm what clients and HDFS NameNodes are using for their groups mapping:

(1)

Client:

Run this on any host that is part of your cluster:

# grep -B1 -A2 "hadoop.security.group.mapping" /etc/hadoop/conf/core-site.xml

(2)

NameNode:

On your NameNode host, run the following:

# grep -B1 -A2 hadoop.security.group.mapping /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep "\-NAMENODE$"| tail -1`/core-site.xml

We can use the information there to help understand where hdfs is getting its mapping for your users.  The fact the "hdfs" user mapping to something other than "hdfs hadoop" for groups is disconcerting indeed.

I should also note that there is one other thing to consider.  If someone there has added values in the following property, that will create a static mapping that will override any other groups mapping:

hadoop.user.group.static.mapping.overrides

Check your server/client core-site.xml for that just to be case.  By default, it won't appear in core-site.xml

[#7]

@bgooley

Hi

Host

  <property>

    <name>hadoop.security.group.mapping</name>

    <value>org.apache.hadoop.security.LdapGroupsMapping</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.url</name>

    <value>ldap://sspeapp01v.sec.oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.user</name>

    <value>cn=admin,dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.base</name>

    <value>dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

    <value>(objectClass=posixAccount)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

    <value>(objectClass=posixGroup)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

    <value>memberUid</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

    <value>cn</value>

  </property>

NameNode

<property>

    <name>hadoop.security.group.mapping</name>

    <value>org.apache.hadoop.security.LdapGroupsMapping</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.url</name>

    <value>ldap://sspeapp01v.sec.oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.user</name>

    <value>cn=admin,dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.password</name>

    <value>********</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.base</name>

    <value>dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

    <value>(objectClass=posixAccount)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

    <value>(objectClass=posixGroup)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

    <value>memberUid</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

    <value>cn</value>

  </property>

And there is no hadoop.user.group.static.mapping.overrides in core-site.xml

[#8]

@Paulina,

It appears that your user search filter may need adjustment.

You have (objectclass=posixAccount) but are missing the part of the filter that accepts the uid.

You might try the following in Hadoop User Group Mapping LDAP User Search Filter:

(&(objectClass=posixAccount)(uid={0}))

I don't have time at the moment to go through the code more and figure out why you are seeing this exact behavior, but I think the above change is a good start.

[#9]

@bgooley

Unfortunately that did not help

[#10]

@Paulina,

The next step, then, is to find out what LDAP commands are being issued on the LDAP server.  Since you are using a non-AD server, your LDAP access logs should give us that information.  I would tail the ldap server access logs while doing the "hdfs groups hdfs" command for instance.

Once we see what ldap commands are run, we should have a better idea of why we are seeing the results we are seeing.

Also, you could use Wireshark or tcpdump to capture packets and view them in Wireshark.  Wireshark decodes LDAP packets so we can actually view the complete conversation between client and server.

[#11]

@bgooley

 Hello!

The changes you have adviced in search filter  (&(objectClass=posixAccount)(uid={0})) had an effect!

Now the command hdfs groups hdfs returns hdfs : hdfs

hdfs groups maslova
maslova : maslova

Now the search bind returns the group, which is in gidNumber of the account.

LDAP log:

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixAccount)(uid=maslova))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (uid) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SEARCH RESULT tag=101 err=0 nentries=1 text=

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=1004)))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (gidNumber) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (memberUid) not indexed

HDFS configuration looks like:

Hadoop User Group Mapping LDAP Group Search Filter (objectClass=posixGroup)

Hadoop User Group Mapping LDAP Group Membership Attribute memberUid

Hadoop User Group Mapping LDAP Group Name Attribute cn

Something is wrong with these lines, I think

[#12]

The reason of this problems was that HDFS is looking for uidNumber in memberUid and Hue is looking for uid in memberUid. As I added uidNumber in memberUid  for my user in addition to other memberUids everything started to work.

On the picture memberUid=1004 and memberUid=maslova is the same user

Is there any other method to fix the problem, because if we leave membership like this we would have to add users in groups twice: uids and uidnumbers