In our case it was a problem with hadoop classpath configuration.
A quick fix will be (a big thank to one of my collegues):
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
You can change it in spark env script or just add the line to your .bashrc.
Thanks for the suggestion.
I have finally reinstalled CDH 5.5 from scratch i.e uninstalled and did a clean installed instead of update. That worked. I am not able to run everything fine.
I was getting the same stacktrace. Apparently some environment variables are wrongly set. Curiously sourcing the spark-env.sh manually to get the (what I think) good environment variables before launching spark-submit worked:
Ok this worked for me too.
Now I am wondering how to really fix this, rather that having to use this workaround everytime.
Any ideas and explanations are welcome
I had the same issue when executing spark-submit. The issue was that I did not have all my paths set and there is an easy way to fix if you know which jars you want to include, simply use the --jars parameter, like this:
spark-submit ... --jars /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars/hadoop-common-2.6.0-cdh5.10.0.jar ...
Another cause of this error is trying to run spark-submit when you only have Spark 2 installed, in which case you just need to run spark2-submit (no configuration changes needed)