- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
The application won't work without a running HiveServer2 and Access denied erors after enabling LDAP
- Labels:
-
Apache Hive
-
Cloudera Hue
Created on ‎09-05-2018 12:35 AM - edited ‎09-16-2022 06:40 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Cloudera Express 5.9.3
Hue 3.11
Sentry is enabled, admin groups are added to Admin Groups and admin users are added into Allowed Connecting Users in the Setry configuration.
SSSD is enabled on the servers, this means that users exist only in LDAP, but can log in into OS using their LDAP credentials. Admin users accounts exist in servers OS and match admin users accounts made in LDAP (and also match admin users accounts in Hue).
System account such as hue, hive, cloudera-scm etc aslo made in LDAP.
After enabling LDAP for Hue adn Hive I recieve errors
Potential misconfiguration detected. Fix and restart Hue.
Hive | The application won't work without a running HiveServer2. |
When trying to enter Security - Hive tables an error Access denied to user occure.
Query Editors- Hive Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user) (code THRIFTTRANSPORT): TTransportException('Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user)',)
Hive log:
org.apache.thrift.transport.TTransportException: PLAIN auth failed: LDAP Authentication failed for user
I have tried all the advices, posted here, nothing helped.
Created ‎10-09-2018 11:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Long journey, so I hope this does the trick.
While not exposed in Cloudera Manager, you can identify an alternative userid attribute. By default, for posix objecdts, the default is memberUid (which won't work for you unless memberUid contains the numeric id).
Try this:
- In Cloudera Manager, navigate to:
Clusters --> HDFS --> Configuration
- Search for:
Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml
- Add the following:
Name: hadoop.security.group.mapping.ldap.posix.attr.uid.name
Value: uid
- Save
- Restart the cluster (so the servers can detect the new settings)
BACKGROUND:
As you observed, by default HDFS assumes that the memberUid attribute will have a value of the uidNumber of the user account. The configuration I mentioned above lets you adjust this so that rather than searching for "memberUid=1004" the search will contain "uid=maslova"
You can find more information here:
This is the description:
"The attribute of posixAccount to use when groups for membership. Mostly useful for schemas wherein groups have memberUids that use an attribute other than uidNumber."
As you can see, the configuration for hdfs meets the needs of your situation.
Cheers,
Ben
Created ‎10-09-2018 11:24 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry for the mistake... I meant that now with the suggested change, the resulting group lookup filter will be:
(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=maslova)))
This is because the hadoop.security.group.mapping.ldap.posix.attr.uid.name property tells ldapgroupmapping which user attribute to use to obtain the value for group lookup. In your LDAP config that is "uid"
Created ‎09-18-2018 04:06 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What I found in the log of Sentry
DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [polymatica]
But why??????
The user maslova in not in group polymatica neither in LDAP nor in OS.
All the mistakes disappear as I add group polymatica in Sentry admins, and in garnts for hive tables.
What is wrong?
Created ‎09-20-2018 10:31 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I agree that this is pretty odd, but we'll figure it out
We see that you have LDAP Groups Mapping configured for HDFS
I assume, then, that HDFS stuff appears to be working correctly.
Let's confirm by running:
# hdfs groups maslova
# hdfs groups hdfs
If Sentry is using the same lookup as HDFS, then we would assume the results would be the same and you would see that maslova is a member of ploymatica group.
However, if the "hdfs groups" command results differ, this indicates that Sentry is not using the same configuration for group lookups.
Let's start by getting those results.
Created ‎09-20-2018 11:55 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello, bgooley!
hdfs groups maslova
maslova : polymatica
hdfs groups hdfs
hdfs : polymatica
Why does it happen? And how to fix it?
Thank you for your answers!
Created ‎10-03-2018 11:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What the result is telling you is that the the configuration you have for groups mapping in HDFS is returning that result. Based on your LDAP and passwd output, it appears that your cluster is using a different means to derive group membership that you thought.
Let's check the following to confirm what clients and HDFS NameNodes are using for their groups mapping:
(1)
Client:
Run this on any host that is part of your cluster:
# grep -B1 -A2 "hadoop.security.group.mapping" /etc/hadoop/conf/core-site.xml
(2)
NameNode:
On your NameNode host, run the following:
# grep -B1 -A2 hadoop.security.group.mapping /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep "\-NAMENODE$"| tail -1`/core-site.xml
We can use the information there to help understand where hdfs is getting its mapping for your users. The fact the "hdfs" user mapping to something other than "hdfs hadoop" for groups is disconcerting indeed.
I should also note that there is one other thing to consider. If someone there has added values in the following property, that will create a static mapping that will override any other groups mapping:
hadoop.user.group.static.mapping.overrides
Check your server/client core-site.xml for that just to be case. By default, it won't appear in core-site.xml
Created ‎10-04-2018 12:35 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi
Host
<property>
<name>hadoop.security.group.mapping</name>
<value>org.apache.hadoop.security.LdapGroupsMapping</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.url</name>
<value>ldap://sspeapp01v.sec.oteco</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.bind.user</name>
<value>cn=admin,dc=sec,dc=oteco</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.base</name>
<value>dc=sec,dc=oteco</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.filter.user</name>
<value>(objectClass=posixAccount)</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.filter.group</name>
<value>(objectClass=posixGroup)</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.attr.member</name>
<value>memberUid</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>
<value>cn</value>
</property>
NameNode
<property>
<name>hadoop.security.group.mapping</name>
<value>org.apache.hadoop.security.LdapGroupsMapping</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.url</name>
<value>ldap://sspeapp01v.sec.oteco</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.bind.user</name>
<value>cn=admin,dc=sec,dc=oteco</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.bind.password</name>
<value>********</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.base</name>
<value>dc=sec,dc=oteco</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.filter.user</name>
<value>(objectClass=posixAccount)</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.filter.group</name>
<value>(objectClass=posixGroup)</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.attr.member</name>
<value>memberUid</value>
</property>
<property>
<name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>
<value>cn</value>
</property>
And there is no hadoop.user.group.static.mapping.overrides in core-site.xml
Created ‎10-05-2018 11:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It appears that your user search filter may need adjustment.
You have (objectclass=posixAccount) but are missing the part of the filter that accepts the uid.
You might try the following in Hadoop User Group Mapping LDAP User Search Filter:
(&(objectClass=posixAccount)(uid={0}))
I don't have time at the moment to go through the code more and figure out why you are seeing this exact behavior, but I think the above change is a good start.
Created ‎10-08-2018 12:01 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎10-08-2018 09:46 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The next step, then, is to find out what LDAP commands are being issued on the LDAP server. Since you are using a non-AD server, your LDAP access logs should give us that information. I would tail the ldap server access logs while doing the "hdfs groups hdfs" command for instance.
Once we see what ldap commands are run, we should have a better idea of why we are seeing the results we are seeing.
Also, you could use Wireshark or tcpdump to capture packets and view them in Wireshark. Wireshark decodes LDAP packets so we can actually view the complete conversation between client and server.
Created ‎10-09-2018 12:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello!
The changes you have adviced in search filter (&(objectClass=posixAccount)(uid={0})) had an effect!
Now the command hdfs groups hdfs returns hdfs : hdfs
hdfs groups maslova
maslova : maslova
Now the search bind returns the group, which is in gidNumber of the account.
LDAP log:
Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixAccount)(uid=maslova))"
Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH attr=cn uidNumber gidNumber
Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (uid) not indexed
Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SEARCH RESULT tag=101 err=0 nentries=1 text=
Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=1004)))"
Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH attr=cn uidNumber gidNumber
Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (gidNumber) not indexed
Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (memberUid) not indexed
HDFS configuration looks like:
Hadoop User Group Mapping LDAP Group Search Filter (objectClass=posixGroup)
Hadoop User Group Mapping LDAP Group Membership Attribute memberUid
Hadoop User Group Mapping LDAP Group Name Attribute cn
Something is wrong with these lines, I think
Created ‎10-09-2018 02:50 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The reason of this problems was that HDFS is looking for uidNumber in memberUid and Hue is looking for uid in memberUid. As I added uidNumber in memberUid for my user in addition to other memberUids everything started to work.
On the picture memberUid=1004 and memberUid=maslova is the same user
Is there any other method to fix the problem, because if we leave membership like this we would have to add users in groups twice: uids and uidnumbers
