Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka to Spark Streaming Issue

Kafka to Spark Streaming Issue

New Contributor

Hi

After successfully completing the Spark tutorial at https://hortonworks.com/hadoop-tutorial/introduction-spark-streaming/, I tried using this code to pass messages from Kafka to Spark Streaming. Maven was given the dependency for spark-streaming-kafka_2.10 as given in the Spark Kafka integration guide. Maven was compiled using command: mvn install clean

When I run the following command,

/bin/spark-submit --verbose  --class JavaDirectKafkaWordCount source.java 192.168.136.128:6667 trucks_events

I get the error : java.lang.ClassNotFoundException: JavaDirectKafkaWordCount

This class is quite clearly there in the source code. I am using the latest Hortonworks Sandbox.

Please help me out.

Thanks

Best Regards,

Adeel

1 REPLY 1

Re: Kafka to Spark Streaming Issue

Mentor

specify the full class including package name, I.e. and source.java should be the jar source.jar, you can find it in target directory after you compile with mvn.

org.apache.spark.examples.SparkPi

#example

spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10
Don't have an account?
Coming from Hortonworks? Activate your account here