New Contributor
Posts: 4
Registered: ‎12-27-2016

Error initializing SparkContext


After I run that following command in Terminal I get an error that I can't solve.

Any ideas? tried to restart the Spark & Yarn services but didn't help.


[cloudera@quickstart ~]$ spark-shell --master yarn-client

ERROR spark.SparkContext: Error initializing SparkContext. File file:/user/spark/applicationHistory does not exist

Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: Error initializing SparkContext

I figured it had to do with history server but haven't dealt with this specifically. Never fear StackOverflow is here.0

The short is that the history server is enabled but the path it is trying to write to is not valid. Either give it a valid path or turn it off. I'd give it a valid path as it is how the UI contains all the wonderful information.

spark.eventLog.dir=/user/spark/applicationHistory (make sure it exist and the spark user as r/w access)