Support Questions

Find answers, ask questions, and share your expertise

spark-submit of spark streaming application is failing in HDP 2.6.2

Hi, My spark-submit command is failing with this error,

Diagnostics: Path /user/admin/.sparkStaging/application_1507577519770_0011/spark-streaming_2.11- does not exist

My spark-submit command is,

spark-submit --jars /home/admin/kafka_2.10-,/home/admin/spark-streaming-kafka-0-8_2.11-2.1.0.jar,/usr/hdp/ KafkaMain --master yarn --deploy-mode cluster --num-executors 100 --driver-memory 1g --executor-memory 1g --verbose --supervise file:///home/cancunadmin/sparkhdfs.jar


Expert Contributor
@Krish Khambadkone - I noticed in the command that you posted that...
spark-streaming_2.11- KafkaMain

has no space between "jar" and "--class"... I assume this was maybe a typo, but I thought I would mention it.

Can you run one of the Spark examples successfully? (e.g. try org.apache.spark.examples.streaming.JavaNetworkWordCount from /usr/hdp/current/spark-client/lib/spark-examples-*.jar (you will need to double check for the exact jar name). If that fails, I recommending sharing the YARN applications logs (i.e. run "yarn logs -applicationId <app id>").

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.