question Re: spark sql is not working on CDH5.3 in Archives of Support Questions (Read Only)

spark sql is not working on CDH5.3

JPMChaseBigData — Fri, 16 Sep 2022 09:22:23 GMT

We upgraded to CDH5.3. spark-sql is not working. Could any bodyk please provide us am I missing any steps we need to follow to spark-sql to work. I'm getting the below error.

Steps we did:

1.copy the hive-site.xml to /etc/spark/conf

2.Try to start the thrifiserver but getting the error.

/opt/cloudera/parcels/CDH/lib/spark/sbin/start-thriftserver.sh
starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /var/log/spark/spark-root-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-rkqcrl-odshaun01.out
failed to launch org.apache.spark.sql.hive.thriftserver.HiveThriftServer2:
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 18 more

bash-4.1$ ./spark-sql
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliDriver
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.cli.CliDriver
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 18 more

Regards,

Venky

Re: spark sql is not working on CDH5.3

srowen — Mon, 23 Feb 2015 18:58:17 GMT

So, Spark SQL is shipped unchanged from upstream. It should mostly work as-is, as a result. It is not formally supported, as it's still an alpha component. Here in particular, have a look at other threads on this forum. I think the issue is that Spark SQL is not yet compatible with the later version of Hive in CDH, so it's not built with Hive support. Some of it should still work, but you have to add the Hive JARs to the classpath at least.

Re: spark sql is not working on CDH5.3

JPMChaseBigData — Mon, 23 Feb 2015 19:26:25 GMT

Thanks for quick reply. I have added the path too. But still not working. correct me if I'm wrong or missing something.

#!/bin/bash
export JAVA_HOME=/usr/java/jdk1.7.0_55
SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark/

SPARK_CLASSPATH=/opt/cloudera/parcels/CDH/lib/spark/lib/*.jar

JARS=""
for j in `ls /opt/cloudera/parcels/CDH/lib/hadoop/client/*.jar`
do
JARS=$JARS:$j
JARS1=$j,$JARS1
done

CLI=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-cli-0.13.1-cdh5.3.1.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/hive-common-0.13.1-cdh5.3.1.jar:=/opt/cloudera/parcels/CDH/lib/hive/lib/hive-jdbc-0.13.1-cdh5.3.1.jar:/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-0.13.1-cdh5.3.1.jar
$SPARK_CLASSPATH:$JARS:$CLI

$SPARK_HOME/bin/spark-sql --master local

Re: spark sql is not working on CDH5.3

srowen — Mon, 23 Feb 2015 19:28:54 GMT

You're probably beyond my knowledge. But the immediate error is easy enough to understand; it can't find the Hive classes, so something is still wrong there. I see a typo in your path for example; there are two jars separated by ":=" Is it just that?

Re: spark sql is not working on CDH5.3

JPMChaseBigData — Mon, 23 Feb 2015 20:03:32 GMT

my bad. I fixed the typo. But still no luck. Thanks.

Re: spark sql is not working on CDH5.3

zhefeng — Tue, 24 Mar 2015 17:52:12 GMT

i have the same issue. Even have hive classpath included, it still gives error.

Anyone has luck on this?

Re: spark sql is not working on CDH5.3

JPMChaseBigData — Tue, 24 Mar 2015 18:32:31 GMT

Hi,

There is bug in the classpath.

You need add a line int the compute-classpath.sh CLASSPATH="$CLASSPATH:/opt/cloudera/parcels/CDH/lib/hive/lib/*" . Then it will work without any issues.

Regards,

Venkat

Re: spark sql is not working on CDH5.3

zhefeng — Wed, 25 Mar 2015 16:35:45 GMT

thanks a lot! this does the trick:)

Re: spark sql is not working on CDH5.3

srowen — Wed, 25 Mar 2015 16:48:48 GMT

To add a little color, yes you can do that, although the CLASSPATH intentionally does not include Hive, since as I understand, Spark doesn't work with the later versions of Hive that CDH 5.3 and beyond use. It still may work enough to do what you need, so, have at it. But you may hit some incompatibilities.

Re: spark sql is not working on CDH5.3

JPMChaseBigData — Wed, 25 Mar 2015 19:37:29 GMT

I agree. What is the best solution for this?

Re: spark sql is not working on CDH5.3

miro — Wed, 03 Jun 2015 07:55:18 GMT

Hi,
Can you write what version of CDH, Spark and Hive working properly?

Miro

Re: spark sql is not working on CDH5.3

diwan.chirayu1367174791 — Mon, 26 Oct 2015 19:27:11 GMT

So finally what version of CDH will Spark & Hive perfectly together?