Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Exception when running maven project for streaming data from Kafka

Highlighted

Exception when running maven project for streaming data from Kafka

New Contributor
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Product$class
	at org.apache.spark.SparkConf$DeprecatedConfig.<init>(SparkConf.scala:723)
	at org.apache.spark.SparkConf$.<init>(SparkConf.scala:571)
	at org.apache.spark.SparkConf$.<clinit>(SparkConf.scala)
	at org.apache.spark.SparkConf.set(SparkConf.scala:92)
	at org.apache.spark.SparkConf.set(SparkConf.scala:81)
	at org.apache.spark.SparkConf.setMaster(SparkConf.scala:113)
	at com.jesperdj.example.HelloScala$.main(HelloScala.scala:11)
	at com.jesperdj.example.HelloScala.main(HelloScala.scala)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 8 more
Given above is the error when running this code
---------------------------------------------------------------------------------------------
package com.jesperdj.example
import org.apache.spark._
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
 
import org.apache.spark.streaming.kafka.KafkaUtils
object HelloScala  {
  def main(args:Array[String])
  {
  val conf = new SparkConf().setMaster("local[*]").setAppName("KafkaReceiver");
    val ssc = new StreamingContext(conf, Seconds(10))
    
    val kafkaStream = KafkaUtils.createStream(ssc, "localhost:2181","spark-streaming-consumer-group", Map("KafkaStreaming" -> 5))
//need to change the topic name and the port number accordingly
    
    kafkaStream.print()  //prints the stream of data received
   
    
    ssc.start()
    ssc.awaitTermination()
}
}
--------------------------------------------------------------------------------------------------------
 When running this scala program it causes exception.Below given is the pom file.
pom.xml
<?xml version="1.0" encoding="UTF-8"?> 
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
             http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>


    <groupId>test</groupId>
    <artifactId>spark-kafka</artifactId>
    <version>1.0-SNAPSHOT</version>


    <repositories>
        <repository>
            <id>hortonworks</id>
            <name>hortonworks repo</name>
            <url>http://repo.hortonworks.com/content/repositories/releases/</url>
        </repository>
    </repositories>


    <dependencies>
    
    
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->


<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.2.0</version>
</dependency>




       <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka -->
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.11</artifactId>
    <version>1.4.0</version>
</dependency>


    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>2.3.0</version>
    <scope>provided</scope>
</dependency>


    
        
        <!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
 <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.4</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-xml</artifactId>
        <version>2.11.0-M4</version>
    </dependency>
    </dependencies>
    <build>
        <defaultGoal>package</defaultGoal>
        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
            <resource>
                <directory>src/test/resources</directory>
                <filtering>true</filtering>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-resources-plugin</artifactId>
                <configuration>
                    <encoding>UTF-8</encoding>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.0</version>
                <configuration>
                    <recompileMode>incremental</recompileMode>
                    <args>
                        <arg>-target:jvm-1.7</arg>
                    </args>
                    <javacArgs>
                        <javacArg>-source</javacArg>
                        <javacArg>1.7</javacArg>
                        <javacArg>-target</javacArg>
                        <javacArg>1.7</javacArg>
                    </javacArgs>
                </configuration>
                <executions>
                    <execution>
                        <id>scala-compile</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>


                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>




            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <finalName>uber-${project.artifactId}-${project.version}</finalName>
                </configuration>
            </plugin>


        </plugins>


    </build>
</project>

Can anyone help to figure this out?
3 REPLIES 3
Highlighted

Re: Exception when running maven project for streaming data from Kafka

New Contributor

Hi Amrutha,


Could you share your pom.xml?

Regards,
Ravi

Highlighted

Re: Exception when running maven project for streaming data from Kafka

Hello

we will be discussing how to stream Twitter data using Kafka.

_________________________

Best Assignment Help Australia

Highlighted

Re: Exception when running maven project for streaming data from Kafka

New Contributor

As I am a professional Big Data Developer I can understand that YouTube videos and the tutorials sites can provide a basic fundamental concepts and not more than that. Now a days the competition is very high so the Depth knowledge and Project is mandatory In a Spark interview. assignments help Australia

Don't have an account?
Coming from Hortonworks? Activate your account here