Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark unable to connect Hive database in HDP 3.0.1

Solved Go to solution
Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

HI @Geoffrey Shelton Okot

Following properties i have defined in custom spark2-default configuration.

spark.hadoop.hive.zookeeper.quorum

spark.hadoop.hive.llap.daemon.service.hosts

spark.datasource.hive.warehouse.load.staging.dir

spark.datasource.hive.warehouse.metastoreUri

spark.sql.hive.hiveserver2.jdbc.url

yes we're using kerberized cluster.

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay

In a Kerberized cluster, you MUST add the below parameter

spark.sql.hive.hiveserver2.jdbc.url.principal = $hiveS2@REALM

The above principle you can either copy it from Advanced hive-site hive.server2.authentication.kerberos.principal or get by

$ klist -kt /etc/security/keytabs/hiveserver2.service.keytab

Please check the correct HS2 keytab in /etc/security/keytabs/ that should resolve the issue.

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

@Geoffrey Shelton Okot

I had already defined spark.sql.hive.hiveserver2.jdbc.url.principal=hive/_HOST@TEST.COM in configuration.

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay

What do you mean by "After restart spark service, hive-site.xml to original xml file." Make sure all changes are made through Ambari otherwise it will be overwritten!
Can you give latest status ...

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor
@Geoffrey Shelton Okot

I had copied manually /etc/hive/conf/hive-site.xml to /etc/spark2/conf/ and restarted spark service. After restart /etc/spark2/conf/hive-site.xml changed to previous hive-site.xml which i had replaced.

Latest status is, still not able to see hive database by spark. even i have also added below properties in spark configuration:

spark.sql.hive.hiveserver2.jdbc.url.principal

spark.hadoop.hive.zookeeper.quorum

spark.hadoop.hive.llap.daemon.service.hosts

spark.datasource.hive.warehouse.load.staging.dir

spark.datasource.hive.warehouse.metastoreUri

spark.sql.hive.hiveserver2.jdbc.url

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay

Can you install hive,spark clients on the hive/Spark nodes?

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

@Geoffrey Shelton Okot

Hive and spark client has already installed on hive and spark node.

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor

@Geoffrey Shelton Okot

Could you please confirm do we really need to enable Interactive query? because after enable Interactive query, i'm unable to start interactive query service. Below are the logs:

2019-01-02T08:36:41,455 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:43,462 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:45,469 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:47,476 INFO [main] LlapStatusServiceDriverConsole: LLAP status unknown

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Mentor

@Vinay

Yes, you need to enable Interactive query.

Did you follow these steps LLAP & Interactive query

Remember also to enable YARN pre-emption via YARN config

HTH

Highlighted

Re: Spark unable to connect Hive database in HDP 3.0.1

Expert Contributor
@Geoffrey Shelton Okot

Ohh. I did not enable the pre-emption via yarn config, It is only point which is pending. Rest of part, i have completed.

let me check with enable yarn pre-emption. Will update you once done it.

Don't have an account?
Coming from Hortonworks? Activate your account here