After successfully completing the Spark tutorial at https://hortonworks.com/hadoop-tutorial/introduction-spark-streaming/, I tried using this code to pass messages from Kafka to Spark Streaming. Maven was given the dependency for spark-streaming-kafka_2.10 as given in the Spark Kafka integration guide. Maven was compiled using command: mvn install clean
When I run the following command,
/bin/spark-submit --verbose --class JavaDirectKafkaWordCount source.java 192.168.136.128:6667 trucks_events
I get the error : java.lang.ClassNotFoundException: JavaDirectKafkaWordCount
This class is quite clearly there in the source code. I am using the latest Hortonworks Sandbox.
Please help me out.
specify the full class including package name, I.e. and source.java should be the jar source.jar, you can find it in target directory after you compile with mvn.
org.apache.spark.examples.SparkPi #example spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10