Reply
Explorer
Posts: 16
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

What I found in the log of Sentry

 

DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [polymatica]

 

But why??????

 

The user maslova in not in group polymatica neither in LDAP nor in OS.

1.jpg2.jpg3.jpg4.jpg

 

All the mistakes disappear as I add group polymatica in Sentry admins, and in garnts for hive tables.

 

What is wrong?

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@Paulina,

 

 

I agree that this is pretty odd, but we'll figure it out

 

We see that you have LDAP Groups Mapping configured for HDFS

I assume, then, that HDFS stuff appears to be working correctly.

 

Let's confirm by running:

 

# hdfs groups maslova

# hdfs groups hdfs

 

If Sentry is using the same lookup as HDFS, then we would assume the results would be the same and you would see that maslova is a member of ploymatica group.

However, if the "hdfs groups" command results differ, this indicates that Sentry is not using the same configuration for group lookups.

 

Let's start by getting those results.

 

Explorer
Posts: 16
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Hello, bgooley!

 

hdfs groups maslova
maslova : polymatica

 

hdfs groups hdfs
hdfs : polymatica

 

Why does it happen? And how to fix it?

 

Thank you for your answers!

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@Paulina,

 

What the result is telling you is that the the configuration you have for groups mapping in HDFS is returning that result.  Based on your LDAP and passwd output, it appears that your cluster is using a different means to derive group membership that you thought.

 

Let's check the following to confirm what clients and HDFS NameNodes are using for their groups mapping:

 

(1)

 

Client:

 

Run this on any host that is part of your cluster:

 

# grep -B1 -A2 "hadoop.security.group.mapping" /etc/hadoop/conf/core-site.xml

 

(2)

 

NameNode:

 

On your NameNode host, run the following:

 

# grep -B1 -A2 hadoop.security.group.mapping /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep "\-NAMENODE$"| tail -1`/core-site.xml

 

We can use the information there to help understand where hdfs is getting its mapping for your users.  The fact the "hdfs" user mapping to something other than "hdfs hadoop" for groups is disconcerting indeed.

 

I should also note that there is one other thing to consider.  If someone there has added values in the following property, that will create a static mapping that will override any other groups mapping:

 

hadoop.user.group.static.mapping.overrides

 

Check your server/client core-site.xml for that just to be case.  By default, it won't appear in core-site.xml

 

 

Explorer
Posts: 16
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@bgooley

 

Hi

 

Host

 

  <property>

    <name>hadoop.security.group.mapping</name>

    <value>org.apache.hadoop.security.LdapGroupsMapping</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.url</name>

    <value>ldap://sspeapp01v.sec.oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.user</name>

    <value>cn=admin,dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.base</name>

    <value>dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

    <value>(objectClass=posixAccount)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

    <value>(objectClass=posixGroup)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

    <value>memberUid</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

    <value>cn</value>

  </property>

 

NameNode

 

<property>

    <name>hadoop.security.group.mapping</name>

    <value>org.apache.hadoop.security.LdapGroupsMapping</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.url</name>

    <value>ldap://sspeapp01v.sec.oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.user</name>

    <value>cn=admin,dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.bind.password</name>

    <value>********</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.base</name>

    <value>dc=sec,dc=oteco</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.user</name>

    <value>(objectClass=posixAccount)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.filter.group</name>

    <value>(objectClass=posixGroup)</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.member</name>

    <value>memberUid</value>

  </property>

  <property>

    <name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

    <value>cn</value>

  </property>

 

 

And there is no hadoop.user.group.static.mapping.overrides in core-site.xml

 

 

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@Paulina,

 

It appears that your user search filter may need adjustment.

 

You have (objectclass=posixAccount) but are missing the part of the filter that accepts the uid.

You might try the following in Hadoop User Group Mapping LDAP User Search Filter:

 

(&(objectClass=posixAccount)(uid={0}))

 

I don't have time at the moment to go through the code more and figure out why you are seeing this exact behavior, but I think the above change is a good start.

 

Explorer
Posts: 16
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@bgooley

 

Unfortunately that did not help

Posts: 953
Topics: 1
Kudos: 226
Solutions: 121
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@Paulina,

 

The next step, then, is to find out what LDAP commands are being issued on the LDAP server.  Since you are using a non-AD server, your LDAP access logs should give us that information.  I would tail the ldap server access logs while doing the "hdfs groups hdfs" command for instance.

 

Once we see what ldap commands are run, we should have a better idea of why we are seeing the results we are seeing.

 

Also, you could use Wireshark or tcpdump to capture packets and view them in Wireshark.  Wireshark decodes LDAP packets so we can actually view the complete conversation between client and server.

 

 

Explorer
Posts: 16
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@bgooley

 

 Hello!

 

The changes you have adviced in search filter  (&(objectClass=posixAccount)(uid={0})) had an effect!

 

Now the command hdfs groups hdfs returns hdfs : hdfs

 

hdfs groups maslova
maslova : maslova

 

Now the search bind returns the group, which is in gidNumber of the account.

 

LDAP log:

 

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixAccount)(uid=maslova))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (uid) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SEARCH RESULT tag=101 err=0 nentries=1 text=

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=1004)))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (gidNumber) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (memberUid) not indexed

 

 

HDFS configuration looks like:

 

Hadoop User Group Mapping LDAP Group Search Filter (objectClass=posixGroup)

Hadoop User Group Mapping LDAP Group Membership Attribute memberUid

Hadoop User Group Mapping LDAP Group Name Attribute cn

 

Something is wrong with these lines, I think

 

Explorer
Posts: 16
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

The reason of this problems was that HDFS is looking for uidNumber in memberUid and Hue is looking for uid in memberUid. As I added uidNumber in memberUid  for my user in addition to other memberUids everything started to work.

 

On the picture memberUid=1004 and memberUid=maslova is the same user

 

memberuid.jpg

 

Is there any other method to fix the problem, because if we leave membership like this we would have to add users in groups twice: uids and uidnumbers

Announcements