Support Questions
Find answers, ask questions, and share your expertise

unable to compile spark 2.4.0 with cloudera 6.3.2

New Contributor

Hi, 

  Downloaded the cloudera spark from github tag used is cdh6.3.2-release, and tried to compile using the below command:

./dev/make-distribution.sh --name custom-spark --tgz -Phadoop-3.1 -Dhadoop.version=3.0.0-cdh6.3.2 -Phive-thriftserver -Phive -Dhive.version=2.1.1-cdh6.3.2

 

Getting below error while compiling hive-thriftserver:

[error] /home/cloud-user/sparkupgrade/spark-cdh6.3.2-release/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:147: value addToClassPath is not a member of object org.apache.hadoop.hive.ql.exec.Utilities
[error] loader = Utilities.addToClassPath(loader, StringUtils.split(auxJars, ","))
[error] ^
[error] /home/cloud-user/sparkupgrade/spark-cdh6.3.2-release/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:303: type mismatch;
[error] found : org.apache.commons.logging.Log
[error] required: org.slf4j.Logger
[error] private val console = new SessionState.LogHelper(LOG)
[error] ^
[info] org.apache.commons.logging.Log <: org.slf4j.Logger?
[info] false
[error] /home/cloud-user/sparkupgrade/spark-cdh6.3.2-release/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala:35: overloaded method constructor Driver with alternatives:
[error] (x$1: org.apache.hadoop.hive.ql.QueryState,x$2: String,x$3: org.apache.hadoop.hive.ql.hooks.HooksLoader,x$4: org.apache.hadoop.hive.ql.QueryInfo)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.ql.QueryState,x$2: String,x$3: org.apache.hadoop.hive.ql.QueryInfo)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.conf.HiveConf,x$2: org.apache.hadoop.hive.ql.hooks.HooksLoader)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.ql.QueryState,x$2: String)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.conf.HiveConf,x$2: String,x$3: org.apache.hadoop.hive.ql.session.LineageState)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.conf.HiveConf,x$2: org.apache.hadoop.hive.ql.Context,x$3: org.apache.hadoop.hive.ql.session.LineageState)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.conf.HiveConf,x$2: String)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.conf.HiveConf,x$2: org.apache.hadoop.hive.ql.session.LineageState)org.apache.hadoop.hive.ql.Driver <and>
[error] (x$1: org.apache.hadoop.hive.conf.HiveConf)org.apache.hadoop.hive.ql.Driver
[error] cannot be applied to ()
[error] extends Driver
[error] ^
[error] three errors found

.....

 

[INFO] Spark Project Hive Thrift Server ................... FAILURE [02:12 min]

.........

--------------------------

The first error : addToClassPath not found in Utilities.java is because it is removed from hive : 2.1.1-cdh6.3.2

Can some one help me how to get through these errors?

I just tried using the hive version : 2.1.1-cdh6.2.0 as it is having the  addToClassPath method , but i am getting last two errors still exists. Anyhow this is not what i wanted. I have to make it work with hive 2.1.1-cdh6.3.2 only.

 

Thanks

Krupa

 

0 REPLIES 0