Using Hive with PAM Authentication

[#1]

PAM authentication for Hive

In some situations an organization does not have an LDAP server and does not want to use Kerberos for the authentication, since this complicates the integration with third-party software.

A quick alternative is PAM authentication which is in Hive since HDP2.2. It uses the OS users and passwords of the host running the hiveserver2 for authentication.

The examples below are done using HDP 2.2/2.3 and Redhat.

1. Install JPam

- Download JPam library and unzip on the hive host: JPam Libraries

- Copy the .so file into the library path of the hive server.

An easy way to find this is to use

ps -ef | grep hiveserver2

and find the -Djava.library.path variable.

For example copy the .so file to the following folder:

/usr/hdp/2.2.4.2-2/hadoop/lib/native/Linux-amd64-64

2. Make shadow file accessible to hive user

/etc/shadow needs to be readable by hive:

chgrp hive /etc/shadow

chmod 550 /etc/shadow

The wiki entry also says to make /etc/login.defs accessible but this doesn't seem to be necessary for the PAM modules I used.

Wiki entry: Hive Security Wiki

3. Set PAM authentication

In Ambari switch authentication to PAM hive.server2.authentication = PAM

4. Set PAM modules

Different possibilities, what worked for me was login, sshd: hive.server2.authentication.pam.services=login,sshd

5. Restart the hive server

You should now be able to login with username, password of the hiveserver2 host users. No kerberos problems no LDAP connection problems.

Note there are good reasons to use Kerberos or Knox because these support encryption and if you have an LDAP environment this is definitely also a good option. But for a quick authentication configuration in an environment that accesses the server over secure networks PAM is a good option.

6. Getting Hue to run with PAM authentication

Hue 2.6 does not officially support PAM. However LDAP works the same for Hue.

- Make a copy of your hive configuration ( for example into /etc/hue/hive-conf )

- Change the authentication setting in the copy to LDAP

- Point hue to this configuration

- Enter the valid hue user, password as "LDAP user"

[#2]

@Benjamin Leonhardi

I setup the Hiverserver2 authentication with PAM using all 1-5 steps but unfortunately still getting invalid login error while using beeline. I used AMD64 JPAM lib.

My hiveserver2 java lib path is:

-Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native

In HDP 2.3, there is no path: /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64

I copied the .so in /usr/hdp/2.3.4.0-3485/hadoop/lib/native.

[#3]

Hi @Benjamin Leonhardi. thanks for this fantastic article. I just got it implemented. a) Created a new OS user account/password (user1/user1), then tried authentication from beeline, it got failed

b) Tried for an another OS user account/password(hive/hive), it got authenticated for the password set on the OS level. so password validation happens against the pam module. but the same didn't happen for user1/user1. Please let me know, do we need to create the same user account in metastore database as well ? if yes, i'm bit confused.

beeline> !connect jdbc:hive2://node2.cluster.net:10000/hive Connecting to jdbc:hive2://node2.cluster.net:10000/hive Enter username for jdbc:hive2://node2.cluster.net:10000/hive: hive Enter password for jdbc:hive2://node2.cluster.net:10000/hive: **** Connected to: Apache Hive (version 1.2.1000.2.4.0.0-169) Driver: Hive JDBC (version 1.2.1000.2.4.0.0-169) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://node2.cluster.net:10000/hive>

[root@node1 ~]# [root@node1 ~]# beeline WARNING: Use "yarn jar" to launch YARN applications. Beeline version 1.2.1000.2.4.0.0-169 by Apache Hive beeline> !connect jdbc:hive2://node2.cluster.net:10000/hive Connecting to jdbc:hive2://node2.cluster.net:10000/hive Enter username for jdbc:hive2://node2.cluster.net:10000/hive: user1 Enter password for jdbc:hive2://node2.cluster.net:10000/hive: ***** Error: Could not open client transport with JDBC Uri: jdbc:hive2://node2.cluster.net:10000/hive: Peer indicated failure: Error validating the login (state=08S01,code=0) 0:

[#4]

Hello Elan, not sure about the question. PAM will authenticate against any linux user with the linux password. So is user1/user1 a valid linux user? You don't need to do anything with the metastore. The user needs a home directory in hdfs however

[#5]

LD_LIBRARY_PATH=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native
ls -ltr /etc/shadow -r-xr-x--- 1 root hive 1809 Jun 28 09:20 /etc/shadow

Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Peer indicated failure: Error validating the login (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Peer indicated failure: Error validating the login
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:210)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:156)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:142)
        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:207)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1149)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1070)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:980)
        at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:705)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:767)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:485)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:468)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Error validating the login
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:185)
        ... 24 more
Beeline version 1.2.1000.2.4.0.0-169 by Apache Hive

@Benjamin Leonhardi

thanks for the inputs.

Sorry I had to recreate the cluster for some reasons, it took some time on this. yes, the user is already part of the OS (username and password are valid). Still beeline returns the same error message.

Also the user home directory exists in the hdfs filesystem

-

[#6]

Hi, I'm trying to configure this in my ambari 2.5.0.3 with Hive 1.2.1.

When I try to connect to hive using jdbc the following error is thrown:

WARN jdbc.HiveConnection: Failed to connect to localhost:10500 Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10500: Peer indicated failure: Error validating the login (state=08S01,code=0)