question Re: Spark sql and Hive tables in Archives of Support Questions (Read Only)

Spark sql and Hive tables

hogiabao2601 — Fri, 16 Sep 2022 09:14:15 GMT

Hi, I installed Spark 1.1.0 and Hive 0.13, I try to run example code

# sc is an existing SparkContext.
from pyspark.sql import HiveContext
sqlContext = HiveContext(sc)

sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src")

# Queries can be expressed in HiveQL.
results = sqlContext.sql("FROM src SELECT key, value").collect()

so I get error:
Exception in thread "Thread-2" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at py4j.reflection.TypeUtil.getClass(TypeUtil.java:265)
at py4j.reflection.TypeUtil.forName(TypeUtil.java:245)
at py4j.commands.ReflectionCommand.getUnknownMember(ReflectionCommand.java:153)
at py4j.commands.ReflectionCommand.execute(ReflectionCommand.java:82)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more

Everyone can help me?

Re: Spark sql and Hive tables

hogiabao2601 — Fri, 28 Nov 2014 03:20:27 GMT

Please help me!

Re: Spark sql and Hive tables

harsha_v — Fri, 28 Nov 2014 15:02:51 GMT

You need to have the hive client jars in your classpath.

Re: Spark sql and Hive tables

hogiabao2601 — Fri, 28 Nov 2014 16:18:07 GMT

Thanks harsha_v,
but I don't know how to add the hive client jars in my classpath, can u explain clear more?
thank you

Re: Spark sql and Hive tables

shekharreddy — Thu, 22 Jan 2015 13:38:30 GMT

copy ur jars to /usr/lib/spark/assembly/lib folder... then check...it should work

thanks,

Shekhar Reddy.

Re: Spark sql and Hive tables

tarekabouzeid91 — Tue, 17 Feb 2015 14:12:44 GMT

i tried to copy the jar " hive-common-0.13.1-cdh5.3.0.jar " - which contains " org.apache.hadoop.hive.conf.HiveConf " in

" /usr/lib/spark/lib " but it still gives me the error :

java.lang.noclassdeffounderror: org.apache.hadoop.hive.conf.HiveConf

Re: Spark sql and Hive tables

guydou — Wed, 11 Mar 2015 13:30:09 GMT

Hi Tarek,

Did you manage to solve this issue?

I am facing the same issue here

Thanks

Re: Spark sql and Hive tables

tarekabouzeid91 — Wed, 11 Mar 2015 13:52:06 GMT

yes i had to rebuild spark to be compatible with hive

http://spark.apache.org/docs/1.2.0/building-spark.html

this section :

Building With Hive and JDBC Support

Re: Spark sql and Hive tables

guydou — Wed, 11 Mar 2015 13:57:29 GMT

I have spark 1.2 (CDH 5.3.1)

do you think I need also to build myself?

Re: Spark sql and Hive tables

tarekabouzeid91 — Wed, 11 Mar 2015 14:19:40 GMT

that solved my problem , as the version build with cloudera isn't build to hive

Re: Spark sql and Hive tables

srowen — Wed, 11 Mar 2015 14:21:36 GMT

You can also just add the HIve jars to your app classpath.

The catch here is that Spark doesn't quite support the later version of Hive in CDH. This might work for what you're trying to do, but if you build your own, you're building for a slightly different version of Hive than you run here.

Re: Spark sql and Hive tables

guydou — Wed, 11 Mar 2015 14:28:14 GMT

Thanks

Re: Spark sql and Hive tables

Stunos — Fri, 24 Apr 2015 20:34:48 GMT

I'm also seeing this error.

Strangely I am including the jar in the spark submit command:

/usr/bin/spark-submit --class com.mycompany.myproduct.spark.sparkhive.Hive2RddTest --master spark://mycluster:7077 --executor-memory 8G --jars hive-common-0.13.1-cdh5.3.1.jar sparkhive.jar "/home/stunos/hive.json" &

Is this insufficient to add this to the classpath? It has worked for other dependencies so presumably spark copies the dependencies to the other nodes. I am puzzled by this exception.

I can attempt to add this jar to /opt/cloudera/parcels/CDH/spark/lib on each node but at this point it is only a voodoo guess since by my logic the command line argument should have been sufficient.

What do you think? Does this mean I probably have to build Spark?