Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

mode deploy cluster

mode deploy cluster

Explorer

Hello Community,

 

I have installed a cluster with ambari hortonworks.

I specified a spark server and spark nodes 

when I launch this command it is working 

/usr/hdp/3.1.4.0-315/spark2/bin/spark-shell --master yarn --deploy-mode client --jars /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar

However wjeb  I change the cluster mode 

/usr/hdp/3.1.4.0-315/spark2/bin/spark-shell --master yarn --deploy-mode cluster --jars /usr/hdp/3.1.4.0-315/spark2/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar
Error: Cluster deploy mode is not applicable to Spark shells.
Run with --help for usage help or --verbose for debug output

 

Should be something wrong with my conf?

Thanks and Regards

Asma

1 REPLY 1
Highlighted

Re: mode deploy cluster

Super Mentor

@asmarz 

Please try using "spark-submit" instead of "spark-shell" as mentioned in: https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.4/running-spark-applications/content/running_sam...

Example:

# /usr/hdp/current/spark2-client/bin/spark-submit  --class org.apache.spark.examples.SparkPi \
    --master yarn \
    --num-executors 1 \
    --driver-memory 512m \
    --executor-memory 512m \
    --executor-cores 1 \
    --deploy-mode cluster \
    /usr/hdp/current/spark2-client/examples/jars/spark-examples_2.11-2.3.2.3.1.4.0-315.jar 10
Don't have an account?
Coming from Hortonworks? Activate your account here