Support Questions
Find answers, ask questions, and share your expertise

KUDU need rebuild post upgrade of CDH cluster

KUDU need rebuild post upgrade of CDH cluster

Explorer

Hi Team,

 

I am facing an issue post CDH upgrade. I'm getting below error while running an application which was build on previous CDH version(5.16.1). At present I'm using CDH 6.3.3

 

Code causing error:

val kuduContext = new KuduContext(s"$KUDU_MASTER",sc);

kuduContext.upsertRows(dataframe1,s"impala::$DB.table");

 

Error which I'm getting:
20/04/01 12:26:12 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: org.apache.kudu.spark.kudu.KuduContext.upsertRows(Lorg/apache/spark/sql/Dataset;Ljava/lang/String;)V
java.lang.NoSuchMethodError: org.apache.kudu.spark.kudu.KuduContext.upsertRows(Lorg/apache/spark/sql/Dataset;Ljava/lang/String;)V
at com.alight.litespark.cdh_upgrade_testing_dev_1$.main(cdh_upgrade_testing_dev_1.scala:44)
at com.alight.litespark.cdh_upgrade_testing_dev_1.main(cdh_upgrade_testing_dev_1.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)

 

old pom.xml dependency content using which the application was built:

<dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-client</artifactId>
<version>1.7.0-cdh5.16.1</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.kudu</groupId>
<artifactId>kudu-spark2_2.11</artifactId>
<version>1.7.0-cdh5.16.1</version>
<scope>provided</scope>
</dependency>

 

After rebuilding the application with new kudu version dependency the issue got resolved.

Kindly provide the resolution so that we don't have to rebuild previous applications.

This is happening both in cluster and client mode job submission.