question Re: Hive Context in CDH 5.3.x in Archives of Support Questions (Read Only)

Hive Context in CDH 5.3.x

Satya-Kota — Fri, 16 Sep 2022 09:17:40 GMT

We have just started with cloudera. I have a question about spark. Does CDH 5.3.x supports Hive Context?

I have installed CDH 5.3. When I try to get the hive context on scala editor. I get the following error

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

error: bad symbolic reference. A signature in HiveContext.class refers to term hive in package org.apache.hadoop which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling HiveContext.class. error:

Thanks,

Satya K

Re: Hive Context in CDH 5.3.x

GautamG — Mon, 05 Jan 2015 22:13:53 GMT

The simplest first check is to ensure the node is a Hive Gateway. You can
set this role in Cloudera Manager.

Re: Hive Context in CDH 5.3.x

Satya-Kota — Tue, 06 Jan 2015 01:37:34 GMT

Hello Gautam:

Thanks for your reply. I have seen the status, actually hive gateway is added on all the nodes, but it is not getting started on any of the nodes and gateway role status and health is always unknown. Are there any other steps I need to do to get this service up? there were no logs for this service as well

While I was installing the CDH 5.3, all the health checks were passed. One other thing is I have let the installer select all the roles. I didn't select any. Please let me know how can I proceed. Thanks much for your help

Thanks,

Satya K

Re: Hive Context in CDH 5.3.x

Satya-Kota — Tue, 06 Jan 2015 01:46:45 GMT

Also I tried to restart the complete hive service and also this specific role. When I stop this gateway role for all the nodes, it's stopped successfully, but when try to start it again it says "Service must be stopped.", but the service is already stopped. I was trying to search any other blogs/ cloudera community to get a solution. Coudln't find a solution.

Thanks,

Satya Kota

Re: Hive Context in CDH 5.3.x

GautamG — Tue, 06 Jan 2015 01:49:53 GMT

The gateway role isn't something that is started. Setting a gateway role
simply copies hadoop configuration files under /etc//conf. So Hive
gateway means /etc/hive/conf. For YARN it would be /etc/yarn/conf and so on.

I'll let someone else respond about the Spark questions.

Re: Hive Context in CDH 5.3.x

Satya-Kota — Tue, 06 Jan 2015 05:37:09 GMT

Thanks Gautam, Got it.

Hello Guys,

Any one else tried using the hive context in spark installed with CDH 5.3.0? Can you please provide your input on this?

Thanks,

Satya K

Re: Hive Context in CDH 5.3.x

Satya-Kota — Wed, 07 Jan 2015 23:17:11 GMT

Tried few different ways I don't think it works in CDH which is wiered. I have installed plain apcahe spark on a fresh cluster, it all works normal

Re: Hive Context in CDH 5.3.x

GautamG — Wed, 07 Jan 2015 23:24:50 GMT

Spark with CDH 5.3.0 doesn't officially support Hive afaik. Glad you
got it working with plain Apache Spark.

Re: Hive Context in CDH 5.3.x

Satya-Kota — Thu, 08 Jan 2015 00:05:27 GMT

Thanks Gautam for the information. Are there plans for supporting this in future versions?

Does this mean with CDH 5.3.0 as of today we won't get spark-sql on hive, which is a known thing?

Thanks,

Satya K

Re: Hive Context in CDH 5.3.x

GautamG — Thu, 08 Jan 2015 00:09:50 GMT

Hive on Spark in CDH 5.3.0 is not supported but i am not privy to any
timetables. Sorry I can't help in that way. I would suggest that you
subscribe to the Announcements forum so you are updated when a new
version is released.

http://community.cloudera.com/t5/Release-Announcements/bd-p/RelAnnounce

Re: Hive Context in CDH 5.3.x

Satya-Kota — Thu, 08 Jan 2015 00:16:14 GMT

Ok. Thank you !

Re: Hive Context in CDH 5.3.x

srowen — Thu, 08 Jan 2015 10:17:39 GMT

Not supported, but, all of the standard bits are there. It should work just like any other installation. You will probably have to put the Hive jars on your classpath manually, says Marcelo.

Re: Hive Context in CDH 5.3.x

tarekabouzeid91 — Tue, 17 Feb 2015 09:30:36 GMT

can you please tell me how to add the hive jars in the classpath ?

Re: Hive Context in CDH 5.3.x

dice — Tue, 17 Feb 2015 10:34:57 GMT

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_mc_hive_udf.html will help.

Re: Hive Context in CDH 5.3.x

tarekabouzeid91 — Tue, 17 Feb 2015 12:34:57 GMT

okey i will try this thanks so much , but i have another question if you please, i add the external jars i need , all of them work normally , but the

" org.apache.hadoop.hive.conf.HiveConf" which exists in hive-common-0.13.1-cdh5.3.0.jar , it gives the error " calss not found " so why that happens ?

the command i run :

sudo spark-submit --class "WordCount" --master local[*] --jars /usr/local/WordCount/target/scala-2.10/spark-streaming-flume_2.11-1.2.0.jar,/usr/lib/avro/avro-ipc-1.7.6-cdh5.3.0.jar,/usr/lib/flume-ng/lib/flume-ng-sdk-1.5.0-cdh5.3.0.jar,/usr/lib/hive/lib/hive-common-0.13.1-cdh5.3.0.jar,/usr/local/WordCount/target/scala-2.10/spark-hive_2.10-1.2.0-cdh5.3.0.jar /usr/local/WordCount/target/scala-2.10/wordcount_2.10-1.0.jar 127.0.0.1 9999