Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to start spark-sql via CLI on CDH 5.4.2.

Unable to start spark-sql via CLI on CDH 5.4.2.

New Contributor

I am trying to start spark-sql via CLI, but I am facing the error followed.

 

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:538)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

 

 

I have copied hive-site.xml from /etc/hive/conf/hive-site.xml to /usr/lib/spark/conf/. Is there anything else I have to configure other than copying the hive-site.xml to spark conf dirextory?

If so, can anyone please guide me step by step on what configurations changes are needed?

 

CDH version (5.4.2)- No configuration changes are made to the existing pre-built configurations. Used Oracle virtual machine to import the CDH iamge.

Spark version - 1.3.0

 

6 REPLIES 6

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Super Collaborator

Sorry for the late reply,

 

The thrift server can not work in CDH 5.4 with Spark. CDH 5.4 has an incompatible version of Hive (newer) which has changes in the API.

These changes in the API cause the thrift server from Spark to not compile. Until Spark upstream catches up with the Hive release it will not work.

 

There is no workaround for this.

 

Wilfred

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

New Contributor

Hey Wilfred

 

so recent updates like:

 

SPARK - 0.9.0-1.cdh4.6.0.p0.98

CDH 5 - 5.4.5-1.cdh5.4.5.p0.7

 

will not fix this problem ? 

 

 

Best regards

Daniel

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Master Collaborator

No, the problem is that Spark (1.3) can't work with recent Hive versions, and CDH ships pretty new Hive releases. 5.4.x updates are maintenance bug fix updates, but this is something more fundamental that Spark will have to fix in a future release like 1.5.

 

(The Spark 0.9.0 parcel is a very old version for CDH4 and is not relevant to you.)

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

New Contributor

thanks a lot for the quick response sowen

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Master Collaborator
Hi Sean,
is there any update on this in CDH 5.5? Or is it possible to somehow get the spark-sql and thriftserver to work with CDH?

Thanks
Tomas

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Super Collaborator

Spark SQL is supported in CDH 5.5, with some limitations. One of those things that we do not support is the hive thriftserver, see: CDH 5.5 docs

The thriftserver dependency on a version of hive and what we ship in CDH is still a problem.

 

Wilfred