Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In HDP3, Unable to connect Hive from Spark(PySpqrk & Spark-Shell) even though using hive warehouse connector API

In HDP3, Unable to connect Hive from Spark(PySpqrk & Spark-Shell) even though using hive warehouse connector API

Explorer

Environment:

HDP 3.0.1

Ranger Enabled (HDFS, YARN, HIVE, HBASE)

Hive 3.1

Spark 2.3.1

Non Kerberized cluster

AWS EC2 instance(authentication using ssh private key)


I followed from the below urls and configured in custom configuration connect hive from spark..

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_configure_a_s...

we have started pyspark with llap jar and zip file:

pyspark --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar --py-files /usr/hdp/current/hive_warehouse_connector/pyspark_hwc-1.0.0.3.0.0.0-1634.zip

Then i created hivewarehose session:

from pyspark_llap import HiveWarehouseSession
hive = HiveWarehouseSession.session(spark).build()


after creating hivewarehouse session I have give below hive command to check the hive server connection

hive.showDatabases()

I got below error..

hive.describeTable("emp")
	Traceback (most recent call last):
	File "", line 1, in 
	File "/tmp/spark-b1374fa6-2eb8-4363-bbf6-4199925667fc/userFiles-874da928-a8b6-47bd-a127-7ca65d5ff096/pyspark_hwc-1.0.0.3.0.0.0-1634.zip/pyspark_llap/sql/session.py", line 135, in describeTable
	File "/usr/hdp/3.0.0.0-1634/spark2/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
	File "/usr/hdp/3.0.0.0-1634/spark2/python/pyspark/sql/utils.py", line 63, in deco
	return f(*a, **kw)
	File "/usr/hdp/3.0.0.0-1634/spark2/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
	py4j.protocol.Py4JJavaError: An error occurred while calling o70.describeTable.
	: java.lang.RuntimeException: shadehive.org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.execute(HiveWarehouseSessionImpl.java:70)
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.describeTable(HiveWarehouseSessionImpl.java:123)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:745)
	Caused by: shadehive.org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at shadehive.org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:300)
	at shadehive.org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:286)
	at shadehive.org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:324)
	at shadehive.org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
	at shadehive.org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:101)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:198)
	at com.hortonworks.spark.sql.hive.llap.JDBCWrapper.useDatabase(HS2JDBCWrapper.scala:215)
	at com.hortonworks.spark.sql.hive.llap.JDBCWrapper.executeStmt(HS2JDBCWrapper.scala:163)
	at com.hortonworks.spark.sql.hive.llap.DefaultJDBCWrapper.executeStmt(HS2JDBCWrapper.scala)
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.lambda$new$1(HiveWarehouseSessionImpl.java:50)
	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.execute(HiveWarehouseSessionImpl.java:67)
	... 12 more
	Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:335)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
	at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
	at org.apache.hive.service.cli.operation.Operation.run(Operation.java:247)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541)
	at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527)
	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
	at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
	at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
	at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
	at com.sun.proxy.$Proxy66.executeStatementAsync(Unknown Source)
	at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:312)
	at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:562)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
	Caused by: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: Permission denied: user [anonymous] does not have [USE] privilege on [test]
	at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:483)
	at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:1306)
	at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:1070)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:697)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1830)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1777)
	at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1772)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
	at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
	... 26 more

I have cross checked all the user privileges and resource access in ranger all are looking good..

Please check the ablove issue and help us to resolve this.. Thanks in advance..

Regards,

Giridharan c


2 REPLIES 2
Highlighted

Re: In HDP3, Unable to connect Hive from Spark(PySpqrk & Spark-Shell) even though using hive warehouse connector API

New Contributor

same issue

Highlighted

Re: In HDP3, Unable to connect Hive from Spark(PySpqrk & Spark-Shell) even though using hive warehouse connector API

Community Manager

@lymweb As this topic is nearly a year old, you will likely be better served by creating a new topic. This will also provide the opportunity to provide specifics unique to your situation. 


Cy Jervis, Community Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum
Don't have an account?
Coming from Hortonworks? Activate your account here