Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to resolve spark thrift server for Hadoop 2.6.0-cdh5.4.0

how to resolve spark thrift server for Hadoop 2.6.0-cdh5.4.0

Contributor

Hi All,

 

I tried to start spark thrift server, but it failed with below error.

 

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:538)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
You need to build Spark with -Phive and -Phive-thriftserver.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

 

For this error, I came across http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/CDH-5-4-2-1-Spark-Thrift-Server-Err... this link. Can you please suggest on the option Like the error suggests, you'll need to manually build a CDH spark source tar-ball with the mentioned flags, to obtain it. The 5.4.2 sources for Spark are at http://archive.cloudera.com/cdh5/cdh/5/spark-1.3.0-cdh5.4.2-src.tar.gz for my CDH version please?

 

Thanks,

Priya

1 REPLY 1
Highlighted

Re: how to resolve spark thrift server for Hadoop 2.6.0-cdh5.4.0

Rising Star

Hello Priya,

 

Thank you for posting your query with Cloudera community.

 

As recommended in your reference post, CDH does not ship with spark thrift server

 

But you could try downloading the source and build with the flags mentioned 

 

[ https://spark.apache.org/docs/latest/building-spark.html

 

Thanks,
Satz