question Re: The application won't work without a running HiveServer2 and Access denied erors after enabling in Support Questions

The application won't work without a running HiveServer2 and Access denied erors after enabling LDAP

Paulina — Fri, 16 Sep 2022 13:40:28 GMT

Cloudera Express 5.9.3

Hue 3.11

Sentry is enabled, admin groups are added to Admin Groups and admin users are added into Allowed Connecting Users in the Setry configuration.

SSSD is enabled on the servers, this means that users exist only in LDAP, but can log in into OS using their LDAP credentials. Admin users accounts exist in servers OS and match admin users accounts made in LDAP (and also match admin users accounts in Hue).

System account such as hue, hive, cloudera-scm etc aslo made in LDAP.

After enabling LDAP for Hue adn Hive I recieve errors

Potential misconfiguration detected. Fix and restart Hue.

Hive	The application won't work without a running HiveServer2.

When trying to enter Security - Hive tables an error Access denied to user occure.

Query Editors- Hive Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user) (code THRIFTTRANSPORT): TTransportException('Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user)',)

Hive log:

org.apache.thrift.transport.TTransportException: PLAIN auth failed: LDAP Authentication failed for user

I have tried all the advices, posted here, nothing helped.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Thu, 06 Sep 2018 16:14:37 GMT

@Paulina,

If you want Hue to authenticate to Hive via LDAP, you need to make sure you have configured the username and password of an LDAP user hue can use for auth:

[beeswax]

auth_username=hue

auth_password=<hue_password>

The above assumes you have a user named "hue" in your ldap server who can authenticate.

You would add this to Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

If you want to use kerberos for Hue to authenticate, then do the following:

(1)

In CM > Hive > Configuration > Advanced > Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml

Name: hive.server2.authentication
Value: kerberos

(2)

Save

(3)

Restart Hue Service

Reason: Hue picks up its Hive authentication method from the hive-site.xml that is emitted. If you enable LDAP authentication in Hive, the hive.server2.authentication becomes LDAP which Hue will then try to use. To tell Hue to use kerberos, you can use the above technique.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Mon, 10 Sep 2018 14:44:25 GMT

Thank you for the answer. The application won't work without a running HiveServer2 error disappeared and Hive Query Editor doesn't show any mistakes.

But Sentry still gives Access denied to admin despite the fact that the user admin in in admin group everywhere: LDAP, Hue, OS

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Wed, 12 Sep 2018 14:18:37 GMT

It seems that Sentry does not get the correct group for the user from LDAP.

DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [llord]

...

ERROR org.apache.sentry.provider.db.generic.service.thrift.SentryGenericPolicyProcessor: Access denied to maslova

LDAP groups:

dn: cn=admins,ou=Groups,dc=dom

objectClass: top

objectClass: posixGroup

cn: admins

gidNumber: 1500

memberUid: llord

memberUid: lmaslova

memberUid: maslova

dn: cn=llord,ou=Groups,dc=dom

objectClass: top

objectClass: posixGroup

cn: llord

gidNumber: 1050

memberUid: llord

Why does it happen?

The other part of log looks like this

15:59:30,697 DEBUG org.apache.sentry.hdfs.UpdateForwarder: #### GetAllUpdatesFrom [type=class org.apache.sentry.hdfs.UpdateablePermissions, reqSeqNum=122, lastCommit=121, lastSeen=121, getMaxUpdateLogSize()=17]

2018-09-12 15:59:30,697 DEBUG org.apache.sentry.hdfs.UpdateForwarder: #### GetAllUpdatesFrom [type=class org.apache.sentry.hdfs.UpdateableAuthzPaths, reqSeqNum=354, lastCommit=353, lastSeen=353, getMaxUpdateLogSize()=49]

2018-09-12 15:59:30,697 DEBUG org.apache.sentry.hdfs.SentryHDFSServiceProcessor: #### Updates requested from HDFS [permReq=122, permResp=<>] [pathReq=354, pathResp=<>]

2018-09-12 15:59:30,697 DEBUG org.apache.thrift.transport.TSaslTransport: data length before wrap: 59

2018-09-12 15:59:30,697 DEBUG org.apache.thrift.transport.TSaslTransport: writing data length: 119

2018-09-12 15:59:30,834 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does not contain key

2018-09-12 15:59:30,834 DEBUG org.apache.thrift.transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslServerTransport@68760f85

2018-09-12 15:59:30,872 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status START and payload length 6

2018-09-12 15:59:30,872 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received start message with status START

2018-09-12 15:59:30,872 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received mechanism name 'GSSAPI'

2018-09-12 15:59:30,873 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Start message handled

2018-09-12 15:59:30,873 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 647

2018-09-12 15:59:30,875 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 108

2018-09-12 15:59:30,890 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 0

2018-09-12 15:59:30,890 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 32

2018-09-12 15:59:30,895 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 32

2018-09-12 15:59:30,896 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status COMPLETE and payload length 0

2018-09-12 15:59:30,896 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Main negotiation loop complete

2018-09-12 15:59:30,896 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does contain key org.apache.thrift.transport.TSocket@589a07cc

2018-09-12 15:59:30,900 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: reading data length: 150

2018-09-12 15:59:30,901 DEBUG org.apache.thrift.transport.TSaslTransport: data length after unwrap: 90

2018-09-12 15:59:30,928 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(llord) return [llord]

2018-09-12 15:59:30,937 DEBUG DataNucleus.Persistence: ExecutionContext "org.datanucleus.ExecutionContextThreadedImpl@45ca0e75" opened for datastore "org.datanucleus.store.rdbms.RDBMSStoreManager@7b3d95a6" with txn="org.datanucleus.TransactionImpl@229250c3"

2018-09-12 15:59:30,938 DEBUG DataNucleus.Transaction: Transaction created [DataNucleus Transaction, ID=Xid=��пїЅ, enlisted resources=[]]

2018-09-12 15:59:30,938 DEBUG DataNucleus.Transaction: Transaction begun for ExecutionContext org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 (optimistic=false)

2018-09-12 15:59:30,938 DEBUG DataNucleus.Query: Query "SELECT UNIQUE FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE this.groupName == :groupName FetchPlan [default]" of language "JDOQL" has been run before so reusing existing generic compilation

2018-09-12 15:59:30,938 DEBUG DataNucleus.Query: Query "SELECT UNIQUE FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE this.groupName == :groupName FetchPlan [default]" of language "JDOQL" for datastore "rdbms-postgresql" has been run before so reusing existing datastore compilation

2018-09-12 15:59:30,943 DEBUG DataNucleus.Connection: Connection "com.jolbox.bonecp.ConnectionHandle@6f1f6551" opened with isolation level "read-committed" and auto-commit=false

2018-09-12 15:59:30,943 DEBUG DataNucleus.Transaction: Running enlist operation on resource: org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc, error code TMNOFLAGS and transaction: [DataNucleus Transaction, ID=Xid=��пїЅ, enlisted resources=[]]

2018-09-12 15:59:30,943 DEBUG DataNucleus.Connection: Managed connection org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc is starting for transaction Xid=��пїЅ with flags 0

2018-09-12 15:59:30,943 DEBUG DataNucleus.Connection: Connection added to the pool : org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@79c69080 [conn=com.jolbox.bonecp.ConnectionHandle@6f1f6551, commitOnRelease=false, closeOnRelease=false, closeOnTxnEnd=true] for key=org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 in factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@48684f4a]

2018-09-12 15:59:30,943 DEBUG DataNucleus.Query: JDOQL Query : Executing "SELECT UNIQUE FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE this.groupName == :groupName" ...

2018-09-12 15:59:30,943 DEBUG DataNucleus.Datastore: Closing PreparedStatement "com.jolbox.bonecp.PreparedStatementHandle@5b3901f5"

2018-09-12 15:59:30,944 DEBUG DataNucleus.Datastore.Native: SELECT 'org.apache.sentry.provider.db.service.model.MSentryGroup' AS NUCLEUS_TYPE,"A0"."CREATE_TIME","A0"."GROUP_NAME","A0"."GROUP_ID" FROM "SENTRY_GROUP" "A0" WHERE "A0"."GROUP_NAME" = <'llord'>

2018-09-12 15:59:30,948 DEBUG DataNucleus.Datastore.Retrieve: Execution Time = 4 ms

2018-09-12 15:59:30,948 DEBUG DataNucleus.Query: JDOQL Query : Execution Time = 5 ms

2018-09-12 15:59:30,948 DEBUG DataNucleus.Transaction: Transaction rolling back for ExecutionContext org.datanucleus.ExecutionContextThreadedImpl@45ca0e75

2018-09-12 15:59:30,949 DEBUG DataNucleus.Transaction: Rolling back [DataNucleus Transaction, ID=Xid=��пїЅ, enlisted resources=[org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc]]

2018-09-12 15:59:30,949 DEBUG DataNucleus.Connection: Managed connection org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc is rolling back for transaction Xid=��пїЅ

2018-09-12 15:59:30,951 DEBUG DataNucleus.Connection: Managed connection org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc rolled back connection for transaction Xid=��пїЅ

2018-09-12 15:59:30,951 DEBUG DataNucleus.Connection: Connection "com.jolbox.bonecp.ConnectionHandle@6f1f6551" closed

2018-09-12 15:59:30,951 DEBUG DataNucleus.Connection: Connection removed from the pool : org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@79c69080 [conn=com.jolbox.bonecp.ConnectionHandle@6f1f6551, commitOnRelease=false, closeOnRelease=false, closeOnTxnEnd=true] for key=org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 in factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@48684f4a]

2018-09-12 15:59:30,951 DEBUG DataNucleus.Transaction: Transaction rolled back in 3 ms

2018-09-12 15:59:30,951 DEBUG DataNucleus.Cache: Level 1 Cache cleared

2018-09-12 15:59:30,951 DEBUG DataNucleus.Persistence: ExecutionContext "org.datanucleus.ExecutionContextThreadedImpl@45ca0e75" closed

2018-09-12 15:59:30,951 DEBUG org.apache.thrift.transport.TSaslTransport: data length before wrap: 69

2018-09-12 15:59:30,951 DEBUG org.apache.thrift.transport.TSaslTransport: writing data length: 129

2018-09-12 15:59:31,037 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does not contain key

2018-09-12 15:59:31,037 DEBUG org.apache.thrift.transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslServerTransport@2bb3ecd5

2018-09-12 15:59:31,039 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status START and payload length 6

2018-09-12 15:59:31,039 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received start message with status START

2018-09-12 15:59:31,039 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received mechanism name 'GSSAPI'

2018-09-12 15:59:31,040 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Start message handled

2018-09-12 15:59:31,040 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 593

2018-09-12 15:59:31,041 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 108

2018-09-12 15:59:31,043 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 0

2018-09-12 15:59:31,043 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 32

2018-09-12 15:59:31,044 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status COMPLETE and payload length 32

2018-09-12 15:59:31,045 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status COMPLETE and payload length 0

2018-09-12 15:59:31,045 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Main negotiation loop complete

2018-09-12 15:59:31,045 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does contain key org.apache.thrift.transport.TSocket@49ea2734

2018-09-12 15:59:31,046 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: reading data length: 190

2018-09-12 15:59:31,046 DEBUG org.apache.thrift.transport.TSaslTransport: data length after unwrap: 130

2018-09-12 15:59:31,047 DEBUG DataNucleus.Persistence: ExecutionContext "org.datanucleus.ExecutionContextThreadedImpl@45ca0e75" opened for datastore "org.datanucleus.store.rdbms.RDBMSStoreManager@7b3d95a6" with txn="org.datanucleus.TransactionImpl@37d7d90f"

2018-09-12 15:59:31,047 DEBUG DataNucleus.Transaction: Transaction created [DataNucleus Transaction, ID=Xid=��пїЅ, enlisted resources=[]]

2018-09-12 15:59:31,047 DEBUG DataNucleus.Transaction: Transaction begun for ExecutionContext org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 (optimistic=false)

2018-09-12 15:59:31,048 DEBUG DataNucleus.Query: Query "SELECT FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE :p1.contains(this.groupName) FetchPlan [default]" of language "JDOQL" has been run before so reusing existing generic compilation

2018-09-12 15:59:31,048 DEBUG DataNucleus.Query: JDOQL Query : Compiling "SELECT FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE :p1.contains(this.groupName)" for datastore

2018-09-12 15:59:31,048 DEBUG DataNucleus.Query: Parameter ParameterExpression{p1} is being resolved as a literal, so the query is no longer precompilable

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Thu, 13 Sep 2018 09:32:37 GMT

Also users, which do not have privileges in Security-Hive tables for some hive tabels can select data. There are no error messages in logs about it.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Thu, 13 Sep 2018 19:15:33 GMT

Hi @Paulina,

Sentry will use OS group lookup to resolve group membership. Are you using OS level configuration that maps to LDAP users? Also, being an Admin does not grant any access to servers; rather, it gives grant privlige.

What we really need to do is clearly define what you have configured in terms of grants and user/group membership so we can start to clarify what you expect to have happen and what is actually happening.

- is the grants you have made that you perceive are not having the desired effect

- the result of "id -Gn <user>"

- groups that are listed as Sentry Admin Groups

- test results with beeline (to remove the level of abstraction introduced by Hue).

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Mon, 17 Sep 2018 09:07:01 GMT

Hello!

1. Sentry does not take the correct groups for admin user from LDAP.

LDAP is used everywhere across Hadoop.

Hue -> Configuration

Authentication Backend - desktop.auth.backend.LdapBackend
LDAP URL ldaps://sspeapp01v.sec.oteco
LDAP Search Base dc=sec,dc=oteco
LDAP Bind User Distinguished Name cn=admin,dc=sec,dc=oteco
LDAP Bind Password xxxxxx
LDAP User Filter (objectClass=posixAccount)
LDAP Username Attribute uid
LDAP Group Filter (objectClass=posixGroup)
LDAP Group Name Attribute cn
LDAP Group Membership Attribute memberUid
Use Search Bind Authentication true
LDAP Server CA Certificate also set

Sentry admin groups: hive, impala, hue, solr, admins
Allowed Connecting Users: hive, impala, hue, hdfs, solr, kafka, maslova

id -Gn maslova maslova wheel

LDAP
dn: cn=admins,ou=Groups,dc=dom
objectClass: top
objectClass: posixGroup
cn: admins
gidNumber: 1051

memberUid: maslova

dn: uid=maslova,ou=users,dc=sec,dc=oteco
objectClass: shadowAccount
objectClass: top
objectClass: posixAccount
objectClass: account
cn: maslova
gidNumber: 1004
homeDirectory: /home/maslova
uid: maslova
uidNumber: 1004
loginShell: /bin/bash
shadowLastChange: 17730
shadowMax: 99999
shadowMin: 0
shadowWarning: 7
userPassword:: e01ENX1JQ3k1WXF4WkIxdVdTd2NWTFNOTGNBPT0=

While searching user in LDAP Sentry return group polymatica

dn: cn=polymatica,ou=groups,dc=sec,dc=oteco
objectClass: top
objectClass: posixGroup
cn: polymatica
gidNumber: 1001
memberUid: polymatica

No group polymatica in OS

2. After recreating grants in Sentry the behavior has changed, but still the grants are not having the desired effect. Also external database disappeared.

The path for external database is /user/maslova/

Grants made

Role admins, group admins, server=server1 action=ALL

create table T1; create table T2;

Error while compiling statement: FAILED: SemanticException No valid privileges User maslova does not have privileges for SWITCHDATABASE The required privileges: Server=server1->Db=*->Table=+->Column=*->action=select;Server=server1->Db=*->Table=+->Column=*->action=insert;

From beeline

create table T1 (i int);
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
User maslova does not have privileges for CREATETABLE
The required privileges: Server=server1->Db=default->action=*; (state=42000,code=40000)

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Mon, 17 Sep 2018 17:25:42 GMT

@Paulina,

When you say that you are using LDAP everywhere, does that include LDAP group mapping for HDFS. I wasn't clear on that.

The reason I ask is that the issue described has nothing to do with Hue at all; rather, this is a matter of Hue attempting to map access based on a defined role that is mapped to a group that includes one or more users.

(1)

beeline attempts to create a table as user maslova

(2)

HiveServer2 verifies this action is allowed by evaluating the desired action via Sentry

(3)

Sentry will attempt to see if maslova is a member of an OS group that is a member of a role that is granted the necessary access. When you ran id -Gn we see the result was:

maslova maslova wheel

However, you granted access to the "admins" role, but maslova is not a member of "admins" group that is mapped to the admins role.

RECOMMENDATION:

Make sure that you have set maslova to be a member of a group that is mapped to the role granted all access to "server1".

Also, verify your HDFS level group mapping by checking what value is set in your HDFS Hadoop User Group Mapping Implementation configuration. Sentry will use the OS mapping by default (ShellBasedUnixGroupsMapping).

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Tue, 18 Sep 2018 07:47:19 GMT

Hello.

Thank you for the detailed response.

LDAP group mapping for HDFS is turned on.

Hadoop User Group Mapping Implementation org.apache.hadoop.security.LdapGroupsMapping

etc

Enable Access Control Lists true

Enable Sentry Synchronization true

Sentry Synchronization Path Prefixes /user/hive/warehouse, /user/maslova (external database)

Should group admins be set in OS if LDAP is used on HDFS?

We use SSSD and it is hard to create the same users in OS as we have in LDAP.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Tue, 18 Sep 2018 10:56:02 GMT

We have made OS group admins on all hadoop hosts and included user maslova into it.

id -Gn maslova
maslova wheel admins

But this action had no effect.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Tue, 18 Sep 2018 11:06:03 GMT

What I found in the log of Sentry

DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [polymatica]

But why??????

The user maslova in not in group polymatica neither in LDAP nor in OS.

All the mistakes disappear as I add group polymatica in Sentry admins, and in garnts for hive tables.

What is wrong?

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Thu, 20 Sep 2018 17:31:09 GMT

@Paulina,

I agree that this is pretty odd, but we'll figure it out

We see that you have LDAP Groups Mapping configured for HDFS

I assume, then, that HDFS stuff appears to be working correctly.

Let's confirm by running:

# hdfs groups maslova

# hdfs groups hdfs

If Sentry is using the same lookup as HDFS, then we would assume the results would be the same and you would see that maslova is a member of ploymatica group.

However, if the "hdfs groups" command results differ, this indicates that Sentry is not using the same configuration for group lookups.

Let's start by getting those results.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Fri, 21 Sep 2018 06:55:02 GMT

Hello, bgooley!

hdfs groups maslova
maslova : polymatica

hdfs groups hdfs
hdfs : polymatica

Why does it happen? And how to fix it?

Thank you for your answers!

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Wed, 03 Oct 2018 18:44:03 GMT

@Paulina,

What the result is telling you is that the the configuration you have for groups mapping in HDFS is returning that result. Based on your LDAP and passwd output, it appears that your cluster is using a different means to derive group membership that you thought.

Let's check the following to confirm what clients and HDFS NameNodes are using for their groups mapping:

(1)

Client:

Run this on any host that is part of your cluster:

# grep -B1 -A2 "hadoop.security.group.mapping" /etc/hadoop/conf/core-site.xml

(2)

NameNode:

On your NameNode host, run the following:

# grep -B1 -A2 hadoop.security.group.mapping /var/run/cloudera-scm-agent/process/`ls -lrt /var/run/cloudera-scm-agent/process/ | awk '{print $9}' |grep "\-NAMENODE$"| tail -1`/core-site.xml

We can use the information there to help understand where hdfs is getting its mapping for your users. The fact the "hdfs" user mapping to something other than "hdfs hadoop" for groups is disconcerting indeed.

I should also note that there is one other thing to consider. If someone there has added values in the following property, that will create a static mapping that will override any other groups mapping:

hadoop.user.group.static.mapping.overrides

Check your server/client core-site.xml for that just to be case. By default, it won't appear in core-site.xml

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Thu, 04 Oct 2018 07:35:35 GMT

@bgooley

Host

<name>hadoop.security.group.mapping</name>

<value>org.apache.hadoop.security.LdapGroupsMapping</value>

</property>

<name>hadoop.security.group.mapping.ldap.url</name>

<value>ldap://sspeapp01v.sec.oteco</value>

</property>

<name>hadoop.security.group.mapping.ldap.bind.user</name>

<value>cn=admin,dc=sec,dc=oteco</value>

</property>

<name>hadoop.security.group.mapping.ldap.base</name>

<value>dc=sec,dc=oteco</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.filter.user</name>

<value>(objectClass=posixAccount)</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.filter.group</name>

<value>(objectClass=posixGroup)</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.attr.member</name>

<value>memberUid</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

</property>

NameNode

<name>hadoop.security.group.mapping</name>

<value>org.apache.hadoop.security.LdapGroupsMapping</value>

</property>

<name>hadoop.security.group.mapping.ldap.url</name>

<value>ldap://sspeapp01v.sec.oteco</value>

</property>

<name>hadoop.security.group.mapping.ldap.bind.user</name>

<value>cn=admin,dc=sec,dc=oteco</value>

</property>

<name>hadoop.security.group.mapping.ldap.bind.password</name>

</property>

<name>hadoop.security.group.mapping.ldap.base</name>

<value>dc=sec,dc=oteco</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.filter.user</name>

<value>(objectClass=posixAccount)</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.filter.group</name>

<value>(objectClass=posixGroup)</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.attr.member</name>

<value>memberUid</value>

</property>

<name>hadoop.security.group.mapping.ldap.search.attr.group.name</name>

</property>

And there is no hadoop.user.group.static.mapping.overrides in core-site.xml

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Fri, 05 Oct 2018 18:12:46 GMT

@Paulina,

It appears that your user search filter may need adjustment.

You have (objectclass=posixAccount) but are missing the part of the filter that accepts the uid.

You might try the following in Hadoop User Group Mapping LDAP User Search Filter:

(&(objectClass=posixAccount)(uid={0}))

I don't have time at the moment to go through the code more and figure out why you are seeing this exact behavior, but I think the above change is a good start.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Mon, 08 Oct 2018 07:01:52 GMT

@bgooley

Unfortunately that did not help

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

bgooley — Mon, 08 Oct 2018 16:46:39 GMT

@Paulina,

The next step, then, is to find out what LDAP commands are being issued on the LDAP server. Since you are using a non-AD server, your LDAP access logs should give us that information. I would tail the ldap server access logs while doing the "hdfs groups hdfs" command for instance.

Once we see what ldap commands are run, we should have a better idea of why we are seeing the results we are seeing.

Also, you could use Wireshark or tcpdump to capture packets and view them in Wireshark. Wireshark decodes LDAP packets so we can actually view the complete conversation between client and server.

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Tue, 09 Oct 2018 07:44:21 GMT

@bgooley

Hello!

The changes you have adviced in search filter (&(objectClass=posixAccount)(uid={0})) had an effect!

Now the command hdfs groups hdfs returns hdfs : hdfs

hdfs groups maslova
maslova : maslova

Now the search bind returns the group, which is in gidNumber of the account.

LDAP log:

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixAccount)(uid=maslova))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (uid) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2613 SEARCH RESULT tag=101 err=0 nentries=1 text=

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH base="dc=sec,dc=oteco" scope=2 deref=3 filter="(&(objectClass=posixGroup)(|(gidNumber=1004)(memberUid=1004)))"

Oct 9 10:16:40 sspeapp01v slapd[29022]: conn=88425 op=2614 SRCH attr=cn uidNumber gidNumber

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (gidNumber) not indexed

Oct 9 10:16:40 sspeapp01v slapd[29022]: <= mdb_equality_candidates: (memberUid) not indexed

HDFS configuration looks like:

Hadoop User Group Mapping LDAP Group Search Filter (objectClass=posixGroup)

Hadoop User Group Mapping LDAP Group Membership Attribute memberUid

Hadoop User Group Mapping LDAP Group Name Attribute cn

Something is wrong with these lines, I think

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Paulina — Tue, 09 Oct 2018 09:50:47 GMT

The reason of this problems was that HDFS is looking for uidNumber in memberUid and Hue is looking for uid in memberUid. As I added uidNumber in memberUid for my user in addition to other memberUids everything started to work.

On the picture memberUid=1004 and memberUid=maslova is the same user

Is there any other method to fix the problem, because if we leave membership like this we would have to add users in groups twice: uids and uidnumbers