Silver ⭐️⭐️
Jun 18, 2024 5:38 AM UTC
Question
Spark submit configuration --deploy-mode has been overrided from cluster to client in singel node minikube
- 3 replies
- 5839 views
- Translate All
Hi ,
We are in the process of migrating from YARN to Kubernetes for its benefits and upgrading our Spark version from 2.4.4 to 3.5.1. As part of this transition, we have decided to use Scala version 2.12.18 and have upgraded Java from version 8 to 11. Currently, I am encountering three main issues:
- I am experiencing an ArithmeticException due to long overflow. Could the switch from Java 8 to 11 be causing this issue?
- The deployment mode specified as cluster in the spark-submit command is being overridden to client.
- I am unable to use AWS Hadoop package classes in spark-submit, despite including the jars in the container.
$SPARK_HOME/bin/spark-submit \
--master k8s://$K8S_SERVER \ \
--deploy-mode cluster \
--name testing \
--class dt.cerebrum.iotengine.sparkjobs.streaming \
--conf spark.kubernetes.file.upload.path=s3a://cb-spark/path \
--conf spark.hadoop.fs.s3a.endpoint="http://xxxxxxx.xxx" \
--conf spark.hadoop.fs.s3a.access.key="xxxx" \
--conf spark.hadoop.fs.s3a.secret.key="xxxxxxxxx" \
--conf spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.fast.upload=true \
--conf spark.hadoop.fs.s3a.path.style.access="true" \
s3a://cb-spark/iot_engine.jar
Any assistance you could provide on these issues would be greatly appreciated.
Thank you.

