Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark unable to connect Hive database in HDP 3.0.1

Solved Go to solution
Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay

Yes, you need to enable Interactive query.

Did you follow these steps LLAP & Interactive query

Remember also to enable YARN pre-emption via YARN config

HTH

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor
@Geoffrey Shelton Okot

Ohh. I did not enable the pre-emption via yarn config, It is only point which is pending. Rest of part, i have completed.

let me check with enable yarn pre-emption. Will update you once done it.

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

@Geoffrey Shelton Okot

No luck. Pre-emption is already enabled via yarn config and all other prerequisite has completed. Hive interactive query service is running fine. Still

19/01/03 05:16:45 INFO RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=vinay@TEST.COM (auth:KERBEROS) retries=1 delay=5 lifetime=0 
19/01/03 05:16:47 INFO CodeGenerator: Code generated in 294.781928 ms
19/01/03 05:16:47 INFO CodeGenerator: Code generated in 18.011739 ms
+------------+ |databaseName| +
------------+ | default| +------------+
Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay
So now the interactive query is running fine and it no longer throws errors, except that you can't see the other databases except the "DEFAULT" ?

IN HDP 3.0 spark uses its own separate catalog this should explain why can't see any hive databases. ToYou should use the HiveWarehouseConnector. work with hive databases please follow this documentation Configuring hiveWarehouseConnector

Please revert

HTH


Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

@Geoffrey Shelton Okot

Yes interactive query is running fine.

i have edited below properties in custom spark2-default configuration:

spark.sql.hive.hiveserver2.jdbc.url.principal

spark.hadoop.hive.zookeeper.quorum

spark.hadoop.hive.llap.daemon.service.hosts

spark.datasource.hive.warehouse.load.staging.dir

spark.datasource.hive.warehouse.metastoreUri

spark.sql.hive.hiveserver2.jdbc.url

After taken restart.

run the spark-shell

sql("show databases").show()

still only DEFAULT database is visible.

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

Re: Spark unable to connect Hive database in HDP 3.0.1

Super Collaborator

Hi Vinay,

use the below code to connect hive and list the databases :

spark-shell --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://hiveserverip:10000/" spark.datasource.hive.warehouse.load.staging.dir="/tmp" spark.hadoop.hive.zookeeper.quorum="zookeeperquoremip:2181" --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar

val hive = com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.session(spark).build()

hive.showDatabases().show(100, false)

Reference article

https://github.com/hortonworks-spark/spark-llap/tree/master

View solution in original post

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor
Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay

Nice it worked out but the solution wasn't far!

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

Almost, we had done. Thanks again @Geoffrey Shelton Okot

Don't have an account?
Coming from Hortonworks? Activate your account here