Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark unable to connect Hive database in HDP 3.0.1

avatar
Rising Star

Hi Folks,

Hope all are doing well.!!!

I've upgrade HDP 2.6.5 to HDP 3.0.1.0-187 successfully. now i'm trying to connecting hive datbases using spark-shell, i'm unable to see any hive databases. Even i have copied /etc/hive/conf/hive-site.xml to /etc/spark2/conf/ and restarted spark service. After restart spark service, hive-site.xml to original xml file.

Have there any alternative solution to resolve the issue?

Kindly assist me to fix the issue.

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hi Vinay,

use the below code to connect hive and list the databases :

spark-shell --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://hiveserverip:10000/" spark.datasource.hive.warehouse.load.staging.dir="/tmp" spark.hadoop.hive.zookeeper.quorum="zookeeperquoremip:2181" --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar

val hive = com.hortonworks.spark.sql.hive.llap.HiveWarehouseBuilder.session(spark).build()

hive.showDatabases().show(100, false)

Reference article

https://github.com/hortonworks-spark/spark-llap/tree/master

View solution in original post

33 REPLIES 33

avatar
Community Manager

@sattar As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question. 



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Master Mentor

@Vinay

Can you share your spark code let me test if you set the parameters as above?

Is your cluster kerberized?

avatar
Rising Star

HI @Geoffrey Shelton Okot

Following properties i have defined in custom spark2-default configuration.

spark.hadoop.hive.zookeeper.quorum

spark.hadoop.hive.llap.daemon.service.hosts

spark.datasource.hive.warehouse.load.staging.dir

spark.datasource.hive.warehouse.metastoreUri

spark.sql.hive.hiveserver2.jdbc.url

yes we're using kerberized cluster.

avatar
Master Mentor

@Vinay

In a Kerberized cluster, you MUST add the below parameter

spark.sql.hive.hiveserver2.jdbc.url.principal = $hiveS2@REALM

The above principle you can either copy it from Advanced hive-site hive.server2.authentication.kerberos.principal or get by

$ klist -kt /etc/security/keytabs/hiveserver2.service.keytab

Please check the correct HS2 keytab in /etc/security/keytabs/ that should resolve the issue.

avatar
Rising Star

@Geoffrey Shelton Okot

I had already defined spark.sql.hive.hiveserver2.jdbc.url.principal=hive/_HOST@TEST.COM in configuration.

avatar
Master Mentor

@Vinay

What do you mean by "After restart spark service, hive-site.xml to original xml file." Make sure all changes are made through Ambari otherwise it will be overwritten!
Can you give latest status ...

avatar
Rising Star
@Geoffrey Shelton Okot

I had copied manually /etc/hive/conf/hive-site.xml to /etc/spark2/conf/ and restarted spark service. After restart /etc/spark2/conf/hive-site.xml changed to previous hive-site.xml which i had replaced.

Latest status is, still not able to see hive database by spark. even i have also added below properties in spark configuration:

spark.sql.hive.hiveserver2.jdbc.url.principal

spark.hadoop.hive.zookeeper.quorum

spark.hadoop.hive.llap.daemon.service.hosts

spark.datasource.hive.warehouse.load.staging.dir

spark.datasource.hive.warehouse.metastoreUri

spark.sql.hive.hiveserver2.jdbc.url

avatar
Master Mentor

@Vinay

Can you install hive,spark clients on the hive/Spark nodes?

avatar
Rising Star

@Geoffrey Shelton Okot

Hive and spark client has already installed on hive and spark node.

avatar
Rising Star

@Geoffrey Shelton Okot

Could you please confirm do we really need to enable Interactive query? because after enable Interactive query, i'm unable to start interactive query service. Below are the logs:

2019-01-02T08:36:41,455 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:43,462 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:45,469 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:47,476 INFO [main] LlapStatusServiceDriverConsole: LLAP status unknown