Reply
New Contributor
Posts: 3
Registered: ‎07-08-2015

Unable to start spark-sql via CLI on CDH 5.4.2.

I am trying to start spark-sql via CLI, but I am facing the error followed.

 

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:538)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.
You need to build Spark with -Phive and -Phive-thriftserver.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

 

 

I have copied hive-site.xml from /etc/hive/conf/hive-site.xml to /usr/lib/spark/conf/. Is there anything else I have to configure other than copying the hive-site.xml to spark conf dirextory?

If so, can anyone please guide me step by step on what configurations changes are needed?

 

CDH version (5.4.2)- No configuration changes are made to the existing pre-built configurations. Used Oracle virtual machine to import the CDH iamge.

Spark version - 1.3.0

 

Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Sorry for the late reply,

 

The thrift server can not work in CDH 5.4 with Spark. CDH 5.4 has an incompatible version of Hive (newer) which has changes in the API.

These changes in the API cause the thrift server from Spark to not compile. Until Spark upstream catches up with the Hive release it will not work.

 

There is no workaround for this.

 

Wilfred

dsa
New Contributor
Posts: 2
Registered: ‎08-26-2015

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Hey Wilfred

 

so recent updates like:

 

SPARK - 0.9.0-1.cdh4.6.0.p0.98

CDH 5 - 5.4.5-1.cdh5.4.5.p0.7

 

will not fix this problem ? 

 

 

Best regards

Daniel

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

No, the problem is that Spark (1.3) can't work with recent Hive versions, and CDH ships pretty new Hive releases. 5.4.x updates are maintenance bug fix updates, but this is something more fundamental that Spark will have to fix in a future release like 1.5.

 

(The Spark 0.9.0 parcel is a very old version for CDH4 and is not relevant to you.)

dsa
New Contributor
Posts: 2
Registered: ‎08-26-2015

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

thanks a lot for the quick response sowen

Master
Posts: 430
Registered: ‎07-01-2015

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Hi Sean,
is there any update on this in CDH 5.5? Or is it possible to somehow get the spark-sql and thriftserver to work with CDH?

Thanks
Tomas
Highlighted
Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: Unable to start spark-sql via CLI on CDH 5.4.2.

Spark SQL is supported in CDH 5.5, with some limitations. One of those things that we do not support is the hive thriftserver, see: CDH 5.5 docs

The thriftserver dependency on a version of hive and what we ship in CDH is still a problem.

 

Wilfred