Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In HDP3, Unable to connect Hive from Spark(PySpqrk & Spark-Shell) even though using hive warehouse connector API

Highlighted

In HDP3, Unable to connect Hive from Spark(PySpqrk & Spark-Shell) even though using hive warehouse connector API

New Contributor

Environment:

HDP 3.0.1

Ranger Enabled (HDFS, YARN, HIVE, HBASE)

Hive 3.1

Spark 2.3.1

Non Kerberized cluster

AWS EC2 instance(authentication using ssh private key)


I followed from the below urls and configured in custom configuration connect hive from spark..

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_configure_a_s...

we have started pyspark with llap jar and zip file:

pyspark --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar --py-files /usr/hdp/current/hive_warehouse_connector/pyspark_hwc-1.0.0.3.0.0.0-1634.zip

Then i created hivewarehose session:

from pyspark_llap import HiveWarehouseSession
hive = HiveWarehouseSession.session(spark).build()


after creating hivewarehouse session I have give below hive command to check the hive server connection

hive.showDatabases()

I got below error..

hive.describeTable("emp")
	Traceback (most recent call last):
	File "", line 1, in 
	File "/tmp/spark-b1374fa6-2eb8-4363-bbf6-4199925667fc/userFiles-874da928-a8b6-47bd-a127-7ca65d5ff096/pyspark_hwc-1.0.0.3.0.0.0-1634.zip/pyspark_llap/sql/session.py", line 135, in describeTable
	File "/usr/hdp/3.0.0.0-1634/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
	File "/usr/hdp/3.0.0.0-1634/spark2/python/pyspark/sql/utils.py", line 63, in deco
	return f(*a, **kw)
	File "/usr/hdp/3.0.0.0-1634/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
	py4j.protocol.Py4JJavaError: An error occurred while calling o70.describeTable.
	: java.lang.RuntimeException: shadehive.org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.execute(HiveWarehouseSessionImpl.java:70)
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.describeTable(HiveWarehouseSessionImpl.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:745)
	Caused by: shadehive.org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at shadehive.org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:300)
	at shadehive.org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:286)
	at shadehive.org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:324)
	at shadehive.org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
	at shadehive.org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:101)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
	at com.hortonworks.spark.sql.hive.llap.JDBCWrapper.useDatabase(HS2JDBCWrapper.scala:215)
	at com.hortonworks.spark.sql.hive.llap.JDBCWrapper.executeStmt(HS2JDBCWrapper.scala:163)
	at com.hortonworks.spark.sql.hive.llap.DefaultJDBCWrapper.executeStmt(HS2JDBCWrapper.scala)
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.lambda$new$1(HiveWarehouseSessionImpl.java:50)
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.execute(HiveWarehouseSessionImpl.java:67)
	... 12 more
	Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy66.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
	Caused by: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:483)
	at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:1306)
	at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:1070)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:697)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1830)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1777)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1772)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	... 26 more

I have cross checked all the user privileges and resource access in ranger all are looking good..

Please check the ablove issue and help us to resolve this.. Thanks in advance..

Regards,

Giridharan c