Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error initializing SparkContext

avatar
Explorer

Hi,

After I run that following command in Terminal I get an error that I can't solve.

Any ideas? tried to restart the Spark & Yarn services but didn't help.

 

[cloudera@quickstart ~]$ spark-shell --master yarn-client

ERROR spark.SparkContext: Error initializing SparkContext.
java.io.FileNotFoundException: File file:/user/spark/applicationHistory does not exist

1 REPLY 1

avatar
Champion
I figured it had to do with history server but haven't dealt with this specifically. Never fear StackOverflow is here.0

The short is that the history server is enabled but the path it is trying to write to is not valid. Either give it a valid path or turn it off. I'd give it a valid path as it is how the UI contains all the wonderful information.

spark.eventLog.enabled=true
spark.eventLog.dir=/user/spark/applicationHistory (make sure it exist and the spark user as r/w access)

http://stackoverflow.com/questions/36038188/error-sparkcontext-error-initializing-sparkcontext