I've noticed a version mismatch in the cloudera doc for the spark2 installation.
From this url ; https://www.cloudera.com/documentation/spark2/latest/topics/spark2_packaging.html#packaging
The CSD version (cloudera1) isn't the latest and then fails when installed along with the "cloudera2" parcel.
The exact error I got during spark2 startup : The CSD version (2.0.0.cloudera1) is not compatible with the current installed version.
For thos who encounter the same issue here is the workaround.
Simply get the latest CSD from here : http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.0.0.cloudera2.jar
Remove the old cloudera1 CSD, restart cld manager and reinstall spark2.
A big thanks for pointing that out. We have updated the Spark2 page to include both versions with respective links.
Let us know if you find anything else that needs correcting.
View solution in original post