Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Spark 1.6 pyspark ML limitations for saving model

Rising Star



Its been a whole week, i have been sturggling with default spark 1.6 , I have trained a model using pyspark ML but i am not able save the model, main reason is that save method is missing from library due to older version. 


Then i tried to updgrade it using right instructions, installation gets completed it goes to good health. But the conf folder for spark 2 is always empty i tried to put required stuff there but it doesn't help. I have a deadline coming if any one could help me asap ? please





Expert Contributor

Hello @hadoopNoob


...but the conf folder for spark 2 is always empty 

The symptoms you have shared indicates that the node from where you're trying to run spark2 binaries doesn't have a gateway role. I am assuming that you are using Cloudera Manager to manage your CDH cluster(?) If yes, please see the documentation Step 5b which requires that we configure a gateway role on the host(s) (usually edge node) from where we plan to launch spark2 binaries (like spark2-shell, spark2-submit, pyspark2).





Once you'd added a gateway role, redeploy client configuration which will ensure that conf directory for spark2 is populated with all the required configurations and xml files.


# alternatives --display spark2-conf
spark2-conf - status is auto.
link currently points to /etc/spark2/conf.cloudera.spark2_on_yarn
/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/etc/spark2/conf.dist - priority 10
/etc/spark2/conf.cloudera.spark2_on_yarn - priority 51
Current `best' version is /etc/spark2/conf.cloudera.spark2_on_yarn.


Please do note that Spark2.2 and above requires JDK8, this is stated here in addition to other prerequisites. 

Let us know how it goes. Good Luck!



Rising Star
Alright, but this wasn't the issue. I have fixed it some how. Thanks for your quick reply !
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.