Reply
Contributor
Posts: 106
Registered: ‎08-07-2017

how to resolve spark thrift server for Hadoop 2.6.0-cdh5.4.0

Hi All,

 

I tried to start spark thrift server, but it failed with below error.

 

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:538)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Failed to load main class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.
You need to build Spark with -Phive and -Phive-thriftserver.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/avro-tools-1.7.6-cdh5.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

 

For this error, I came across http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/CDH-5-4-2-1-Spark-Thrift-Server-Err... this link. Can you please suggest on the option Like the error suggests, you'll need to manually build a CDH spark source tar-ball with the mentioned flags, to obtain it. The 5.4.2 sources for Spark are at http://archive.cloudera.com/cdh5/cdh/5/spark-1.3.0-cdh5.4.2-src.tar.gz for my CDH version please?

 

Thanks,

Priya

Cloudera Employee
Posts: 9
Registered: ‎05-15-2018

Re: how to resolve spark thrift server for Hadoop 2.6.0-cdh5.4.0

[ Edited ]

Hello Priya,

 

Thank you for posting your query with Cloudera community.

 

As recommended in your reference post, CDH does not ship with spark thrift server

 

But you could try downloading the source and build with the flags mentioned 

 

[ https://spark.apache.org/docs/latest/building-spark.html

 

Thanks,
Satz
Announcements