Reply
Highlighted
Explorer
Posts: 13
Registered: ‎09-04-2018
Accepted Solution

The application won't work without a running HiveServer2 and Access denied erors after enabling LDAP

Cloudera Express 5.9.3

Hue 3.11

Sentry is enabled, admin groups are added to Admin Groups and admin users are added into Allowed Connecting Users in the Setry configuration.

 

SSSD is enabled on the servers, this means that users exist only in LDAP, but can log in into OS using their LDAP credentials. Admin users accounts exist in servers OS and match admin users accounts made in LDAP (and also match admin users accounts in Hue).

System account such as hue, hive, cloudera-scm etc aslo made in LDAP.

 

After enabling LDAP for Hue adn Hive I recieve errors

Potential misconfiguration detected. Fix and restart Hue.

Hive

The application won't work without a running HiveServer2.

When trying to enter Security - Hive tables an error Access denied to user occure.

 

Query Editors- Hive Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user) (code THRIFTTRANSPORT): TTransportException('Bad status: 3 (PLAIN auth failed: LDAP Authentication failed for user)',)

 

Hive log:

org.apache.thrift.transport.TTransportException: PLAIN auth failed: LDAP Authentication failed for user

 

 

I have tried all the advices, posted here, nothing helped.

Posts: 911
Topics: 1
Kudos: 208
Solutions: 112
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@Paulina,

 

If you want Hue to authenticate to Hive via LDAP, you need to make sure you have configured the username and password of an LDAP user hue can use for auth:

 

[beeswax]

auth_username=hue

auth_password=<hue_password>

 

The above assumes you have a user named "hue" in your ldap server who can authenticate.

You would add this to Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini

 

If you want to use kerberos for Hue to authenticate, then do the following:

 

(1)

 

In CM > Hive > Configuration > Advanced > Hive Client Advanced Configuration Snippet (Safety Valve) for hive-site.xml

 

Name: hive.server2.authentication
Value: kerberos

 

(2)

 

Save


(3)

 

Restart Hue Service

 

Reason:  Hue picks up its Hive authentication method from the hive-site.xml that is emitted.  If you enable LDAP authentication in Hive, the hive.server2.authentication becomes LDAP which Hue will then try to use.  To tell Hue to use kerberos, you can use the above technique.

 

 

Explorer
Posts: 13
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Thank you for the answer. The application won't work without a running HiveServer2 error disappeared and Hive Query Editor doesn't show any mistakes.

 

But Sentry still gives Access denied to admin despite the fact that the user admin in in admin group everywhere: LDAP, Hue, OS

Explorer
Posts: 13
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

[ Edited ]

It seems that Sentry does not get the correct group for the user from LDAP.

 

DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(maslova) return [llord]

...

ERROR org.apache.sentry.provider.db.generic.service.thrift.SentryGenericPolicyProcessor: Access denied to maslova

 

LDAP groups:

 

 

dn: cn=admins,ou=Groups,dc=dom

objectClass: top

objectClass: posixGroup

cn: admins

gidNumber: 1500

memberUid: llord

memberUid: lmaslova

memberUid: maslova

 

dn: cn=llord,ou=Groups,dc=dom

objectClass: top

objectClass: posixGroup

cn: llord

gidNumber: 1050

memberUid: llord

 

Why does it happen?

 

The other part of log looks like this

 

 

15:59:30,697 DEBUG org.apache.sentry.hdfs.UpdateForwarder: #### GetAllUpdatesFrom [type=class org.apache.sentry.hdfs.UpdateablePermissions, reqSeqNum=122, lastCommit=121, lastSeen=121, getMaxUpdateLogSize()=17]

2018-09-12 15:59:30,697 DEBUG org.apache.sentry.hdfs.UpdateForwarder: #### GetAllUpdatesFrom [type=class org.apache.sentry.hdfs.UpdateableAuthzPaths, reqSeqNum=354, lastCommit=353, lastSeen=353, getMaxUpdateLogSize()=49]

2018-09-12 15:59:30,697 DEBUG org.apache.sentry.hdfs.SentryHDFSServiceProcessor: #### Updates requested from HDFS [permReq=122, permResp=<>] [pathReq=354, pathResp=<>]

2018-09-12 15:59:30,697 DEBUG org.apache.thrift.transport.TSaslTransport: data length before wrap: 59

2018-09-12 15:59:30,697 DEBUG org.apache.thrift.transport.TSaslTransport: writing data length: 119

2018-09-12 15:59:30,834 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does not contain key

2018-09-12 15:59:30,834 DEBUG org.apache.thrift.transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslServerTransport@68760f85

2018-09-12 15:59:30,872 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status START and payload length 6

2018-09-12 15:59:30,872 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received start message with status START

2018-09-12 15:59:30,872 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received mechanism name 'GSSAPI'

2018-09-12 15:59:30,873 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Start message handled

2018-09-12 15:59:30,873 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 647

2018-09-12 15:59:30,875 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 108

2018-09-12 15:59:30,890 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 0

2018-09-12 15:59:30,890 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 32

2018-09-12 15:59:30,895 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 32

2018-09-12 15:59:30,896 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status COMPLETE and payload length 0

2018-09-12 15:59:30,896 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Main negotiation loop complete

2018-09-12 15:59:30,896 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does contain key org.apache.thrift.transport.TSocket@589a07cc

2018-09-12 15:59:30,900 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: reading data length: 150

2018-09-12 15:59:30,901 DEBUG org.apache.thrift.transport.TSaslTransport: data length after unwrap: 90

2018-09-12 15:59:30,928 DEBUG org.apache.hadoop.security.LdapGroupsMapping: doGetGroups(llord) return [llord]

2018-09-12 15:59:30,937 DEBUG DataNucleus.Persistence: ExecutionContext "org.datanucleus.ExecutionContextThreadedImpl@45ca0e75" opened for datastore "org.datanucleus.store.rdbms.RDBMSStoreManager@7b3d95a6" with txn="org.datanucleus.TransactionImpl@229250c3"

2018-09-12 15:59:30,938 DEBUG DataNucleus.Transaction: Transaction created [DataNucleus Transaction, ID=Xid=����, enlisted resources=[]]

2018-09-12 15:59:30,938 DEBUG DataNucleus.Transaction: Transaction begun for ExecutionContext org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 (optimistic=false)

2018-09-12 15:59:30,938 DEBUG DataNucleus.Query: Query "SELECT UNIQUE FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE this.groupName == :groupName FetchPlan [default]" of language "JDOQL" has been run before so reusing existing generic compilation

2018-09-12 15:59:30,938 DEBUG DataNucleus.Query: Query "SELECT UNIQUE FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE this.groupName == :groupName FetchPlan [default]" of language "JDOQL" for datastore "rdbms-postgresql" has been run before so reusing existing datastore compilation

2018-09-12 15:59:30,943 DEBUG DataNucleus.Connection: Connection "com.jolbox.bonecp.ConnectionHandle@6f1f6551" opened with isolation level "read-committed" and auto-commit=false

2018-09-12 15:59:30,943 DEBUG DataNucleus.Transaction: Running enlist operation on resource: org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc, error code TMNOFLAGS and transaction: [DataNucleus Transaction, ID=Xid=����, enlisted resources=[]]

2018-09-12 15:59:30,943 DEBUG DataNucleus.Connection: Managed connection org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc is starting for transaction Xid=���� with flags 0

2018-09-12 15:59:30,943 DEBUG DataNucleus.Connection: Connection added to the pool : org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@79c69080 [conn=com.jolbox.bonecp.ConnectionHandle@6f1f6551, commitOnRelease=false, closeOnRelease=false, closeOnTxnEnd=true] for key=org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 in factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@48684f4a]

2018-09-12 15:59:30,943 DEBUG DataNucleus.Query: JDOQL Query : Executing "SELECT UNIQUE FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE this.groupName == :groupName" ...

2018-09-12 15:59:30,943 DEBUG DataNucleus.Datastore: Closing PreparedStatement "com.jolbox.bonecp.PreparedStatementHandle@5b3901f5"

2018-09-12 15:59:30,944 DEBUG DataNucleus.Datastore.Native: SELECT 'org.apache.sentry.provider.db.service.model.MSentryGroup' AS NUCLEUS_TYPE,"A0"."CREATE_TIME","A0"."GROUP_NAME","A0"."GROUP_ID" FROM "SENTRY_GROUP" "A0" WHERE "A0"."GROUP_NAME" = <'llord'>

2018-09-12 15:59:30,948 DEBUG DataNucleus.Datastore.Retrieve: Execution Time = 4 ms

2018-09-12 15:59:30,948 DEBUG DataNucleus.Query: JDOQL Query : Execution Time = 5 ms

2018-09-12 15:59:30,948 DEBUG DataNucleus.Transaction: Transaction rolling back for ExecutionContext org.datanucleus.ExecutionContextThreadedImpl@45ca0e75

2018-09-12 15:59:30,949 DEBUG DataNucleus.Transaction: Rolling back [DataNucleus Transaction, ID=Xid=����, enlisted resources=[org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc]]

2018-09-12 15:59:30,949 DEBUG DataNucleus.Connection: Managed connection org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc is rolling back for transaction Xid=����

2018-09-12 15:59:30,951 DEBUG DataNucleus.Connection: Managed connection org.datanucleus.store.rdbms.ConnectionFactoryImpl$EmulatedXAResource@11cdd1bc rolled back connection for transaction Xid=����

2018-09-12 15:59:30,951 DEBUG DataNucleus.Connection: Connection "com.jolbox.bonecp.ConnectionHandle@6f1f6551" closed

2018-09-12 15:59:30,951 DEBUG DataNucleus.Connection: Connection removed from the pool : org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@79c69080 [conn=com.jolbox.bonecp.ConnectionHandle@6f1f6551, commitOnRelease=false, closeOnRelease=false, closeOnTxnEnd=true] for key=org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 in factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@48684f4a]

2018-09-12 15:59:30,951 DEBUG DataNucleus.Transaction: Transaction rolled back in 3 ms

2018-09-12 15:59:30,951 DEBUG DataNucleus.Cache: Level 1 Cache cleared

2018-09-12 15:59:30,951 DEBUG DataNucleus.Persistence: ExecutionContext "org.datanucleus.ExecutionContextThreadedImpl@45ca0e75" closed

2018-09-12 15:59:30,951 DEBUG org.apache.thrift.transport.TSaslTransport: data length before wrap: 69

2018-09-12 15:59:30,951 DEBUG org.apache.thrift.transport.TSaslTransport: writing data length: 129

2018-09-12 15:59:31,037 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does not contain key

2018-09-12 15:59:31,037 DEBUG org.apache.thrift.transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslServerTransport@2bb3ecd5

2018-09-12 15:59:31,039 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status START and payload length 6

2018-09-12 15:59:31,039 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received start message with status START

2018-09-12 15:59:31,039 DEBUG org.apache.thrift.transport.TSaslServerTransport: Received mechanism name 'GSSAPI'

2018-09-12 15:59:31,040 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Start message handled

2018-09-12 15:59:31,040 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 593

2018-09-12 15:59:31,041 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 108

2018-09-12 15:59:31,043 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status OK and payload length 0

2018-09-12 15:59:31,043 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status OK and payload length 32

2018-09-12 15:59:31,044 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Received message with status COMPLETE and payload length 32

2018-09-12 15:59:31,045 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Writing message with status COMPLETE and payload length 0

2018-09-12 15:59:31,045 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: Main negotiation loop complete

2018-09-12 15:59:31,045 DEBUG org.apache.thrift.transport.TSaslServerTransport: transport map does contain key org.apache.thrift.transport.TSocket@49ea2734

2018-09-12 15:59:31,046 DEBUG org.apache.thrift.transport.TSaslTransport: SERVER: reading data length: 190

2018-09-12 15:59:31,046 DEBUG org.apache.thrift.transport.TSaslTransport: data length after unwrap: 130

2018-09-12 15:59:31,047 DEBUG DataNucleus.Persistence: ExecutionContext "org.datanucleus.ExecutionContextThreadedImpl@45ca0e75" opened for datastore "org.datanucleus.store.rdbms.RDBMSStoreManager@7b3d95a6" with txn="org.datanucleus.TransactionImpl@37d7d90f"

2018-09-12 15:59:31,047 DEBUG DataNucleus.Transaction: Transaction created [DataNucleus Transaction, ID=Xid=����, enlisted resources=[]]

2018-09-12 15:59:31,047 DEBUG DataNucleus.Transaction: Transaction begun for ExecutionContext org.datanucleus.ExecutionContextThreadedImpl@45ca0e75 (optimistic=false)

2018-09-12 15:59:31,048 DEBUG DataNucleus.Query: Query "SELECT FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE :p1.contains(this.groupName) FetchPlan [default]" of language "JDOQL" has been run before so reusing existing generic compilation

2018-09-12 15:59:31,048 DEBUG DataNucleus.Query: JDOQL Query : Compiling "SELECT FROM org.apache.sentry.provider.db.service.model.MSentryGroup WHERE :p1.contains(this.groupName)" for datastore

2018-09-12 15:59:31,048 DEBUG DataNucleus.Query: Parameter ParameterExpression{p1} is being resolved as a literal, so the query is no longer precompilable

Explorer
Posts: 13
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Also users, which do not have privileges in Security-Hive tables for some hive tabels can select data. There are no error messages in logs about it.

Posts: 911
Topics: 1
Kudos: 208
Solutions: 112
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Hi @Paulina,

 

Sentry will use OS group lookup to resolve group membership.  Are you using OS level configuration that maps to LDAP users?  Also, being an Admin does not grant any access to servers; rather, it gives grant privlige.

 

What we really need to do is clearly define what you have configured in terms of grants and user/group membership so we can start to clarify what you expect to have happen and what is actually happening.

 

- is the grants you have made that you perceive are not having the desired effect

- the result of "id -Gn <user>"

- groups that are listed as Sentry Admin Groups

- test results with beeline (to remove the level of abstraction introduced by Hue).

 

 

Explorer
Posts: 13
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Hello!

 

1. Sentry does not take the correct groups for admin user from LDAP.


LDAP is used everywhere across Hadoop.

 

Hue -> Configuration

 

Authentication Backend - desktop.auth.backend.LdapBackend
LDAP URL ldaps://sspeapp01v.sec.oteco
LDAP Search Base dc=sec,dc=oteco
LDAP Bind User Distinguished Name cn=admin,dc=sec,dc=oteco
LDAP Bind Password xxxxxx
LDAP User Filter (objectClass=posixAccount)
LDAP Username Attribute uid
LDAP Group Filter (objectClass=posixGroup)
LDAP Group Name Attribute cn
LDAP Group Membership Attribute memberUid
Use Search Bind Authentication true
LDAP Server CA Certificate also set

 

Sentry admin groups: hive, impala, hue, solr, admins
Allowed Connecting Users: hive, impala, hue, hdfs, solr, kafka, maslova

 

id -Gn maslova maslova wheel

 

LDAP
dn: cn=admins,ou=Groups,dc=dom
objectClass: top
objectClass: posixGroup
cn: admins
gidNumber: 1051

 

memberUid: maslova

dn: uid=maslova,ou=users,dc=sec,dc=oteco
objectClass: shadowAccount
objectClass: top
objectClass: posixAccount
objectClass: account
cn: maslova
gidNumber: 1004
homeDirectory: /home/maslova
uid: maslova
uidNumber: 1004
loginShell: /bin/bash
shadowLastChange: 17730
shadowMax: 99999
shadowMin: 0
shadowWarning: 7
userPassword:: e01ENX1JQ3k1WXF4WkIxdVdTd2NWTFNOTGNBPT0=


While searching user in LDAP Sentry return group polymatica

 

dn: cn=polymatica,ou=groups,dc=sec,dc=oteco
objectClass: top
objectClass: posixGroup
cn: polymatica
gidNumber: 1001
memberUid: polymatica

 

No group polymatica in OS

 

 

2. After recreating grants in Sentry the behavior has changed, but still the grants are not having the desired effect. Also external database disappeared.


The path for external database is /user/maslova/

 

Grants made

Role admins, group admins, server=server1  action=ALL

 

create table T1; create table T2;

 

Error while compiling statement: FAILED: SemanticException No valid privileges User maslova does not have privileges for SWITCHDATABASE The required privileges: Server=server1->Db=*->Table=+->Column=*->action=select;Server=server1->Db=*->Table=+->Column=*->action=insert;

 

From beeline

 

create table T1 (i int);
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
 User maslova does not have privileges for CREATETABLE
 The required privileges: Server=server1->Db=default->action=*; (state=42000,code=40000)

Posts: 911
Topics: 1
Kudos: 208
Solutions: 112
Registered: ‎04-22-2014

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

@Paulina,

 

When you say that you are using LDAP everywhere, does that include LDAP group mapping for HDFS.  I wasn't clear on that.

The reason I ask is that the issue described has nothing to do with Hue at all; rather, this is a matter of Hue attempting to map access based on a defined role that is mapped to a group that includes one or more users.

 

(1)

 

beeline attempts to create a table as user maslova

 

(2)

 

HiveServer2 verifies this action is allowed by evaluating the desired action via Sentry

 

(3)

 

Sentry will attempt to see if maslova is a member of an OS group that is a member of a role that is granted the necessary access.  When you ran id -Gn we see the result was:

 

maslova maslova wheel

 

However, you granted access to the "admins" role, but maslova is not a member of "admins" group that is mapped to the admins role.

 

RECOMMENDATION:

 

Make sure that you have set maslova to be a member of a group that is mapped to the role granted all access to "server1".

 

Also, verify your HDFS level group mapping by checking what value is set in your HDFS Hadoop User Group Mapping Implementation configuration.  Sentry will use the OS mapping by default (ShellBasedUnixGroupsMapping).

 

 

 

Explorer
Posts: 13
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

Hello.

 

Thank you for the detailed response.

 

LDAP group mapping for HDFS is turned on.

 

Hadoop User Group Mapping Implementation org.apache.hadoop.security.LdapGroupsMapping

 

etc

 

Enable Access Control Lists true

Enable Sentry Synchronization true

Sentry Synchronization Path Prefixes     /user/hive/warehouse, /user/maslova (external database)

 

Should group admins be set in OS if LDAP is used on HDFS?

We use SSSD and it is hard to create the same users in OS as we have in LDAP.

Explorer
Posts: 13
Registered: ‎09-04-2018

Re: The application won't work without a running HiveServer2 and Access denied erors after enabling

We have made OS group admins on all hadoop hosts and included user maslova into it.

 

id -Gn maslova
maslova wheel admins

 

But this action had no effect.

 

Announcements