Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Who agreed with this solution

Re: Cloudera Quickstart VM 5.8 - Kafka kafka_2.10-0.8.2.0-kafka-1.4.0.jar - IllegalArgumentException

New Contributor

Finally I am able to resolve this issue - it is more related to improper usage of kafka dependencies in my pom.xml

 

This is what I did :

1. Created a new Cloudera Quickstart VM 5.8

 

2. Installed Kafka by following the commands mentioned in https://www.cloudera.com/documentation/kafka/latest/topics/kafka_packaging.html

 

In the above HTML link read carefully on maven dependency we have to use. Also check below pom.xml where kafka_clients is excluded and added from Cloudera Repository

 

3. In my maven pom.xml I have below configuration

 

Please note below kafka_clients exclusion while defining spark-streaming-kafka

 

 

<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>

 

<properties>

<java.version>1.8</java.version>

<scala.minor.version>2.10</scala.minor.version>

<scala.complete.version>${scala.minor.version}.6</scala.complete.version>

<spark.version>1.6.1</spark.version>

</properties>

.....

 

<dependencies>

...............

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_${scala.minor.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0-kafka-2.0.1</version>
</dependency>

...............

<dependencies>

 

Hope this will help others

 

Thanks,

TrainingMyHobby

 

Who agreed with this solution