02-26-2018 05:37 AM - last edited on 02-26-2018 05:52 AM by cjervis
After going throught the steps of installing the Spark 2.2 on all nodes, I'm getting Unsupported major.minor version 52.0 when running spark2-submit.
The CDH version is 5.12
Java 1.8 is installed in /usr/java/jdk1.8.0_144
All hosts have the Java Home Directory line modified in CM.
JAVA_HOME is set on the nodes where I run the spark2-submit line (it's not set on the worker nodes but I didn't get that it should from the installation steps).
Furthermore, spark2-shell works without issues, gateway roles are set on the nodes.
Spark 2.2.1 stand alone, installed on the same node works without issues, with the same jar that does not work on spark2-submit.
I might have missed something but I checked and I can't find what it might be. If I need to post additional info please let me know. Thank you
03-02-2018 08:52 AM
After a server reboot, spark2-shell doesn't work anymore either, I get the same error.
and now the spark2-shell starts.
If I start spark-shell (1.6.0), it uses java 7
I checked yarn and hdfs running processes and they use java 1.8.0 (the one in the path added in CM) so the setting is correct for other services but I suspect for for spark2-shell and spark2-submit it's not.
Not sure how to proceed, I retraced the official guide to installing spark 2.2 and I can't find something that I missed. Any recommendation would be much appreciated.