After a server reboot, spark2-shell doesn't work anymore either, I get the same error.
and now the spark2-shell starts.
If I start spark-shell (1.6.0), it uses java 7
I checked yarn and hdfs running processes and they use java 1.8.0 (the one in the path added in CM) so the setting is correct for other services but I suspect for for spark2-shell and spark2-submit it's not.
Not sure how to proceed, I retraced the official guide to installing spark 2.2 and I can't find something that I missed. Any recommendation would be much appreciated.