Member since
06-16-2021
2
Posts
2
Kudos Received
0
Solutions
06-16-2021
06:13 AM
1 Kudo
Tested without dynamic allocation, a new container is spawning. Is there an option with dynamic allocation like --num-executors but should be --min-num-executors spark.streaming.dynamicAllocation.minExecutors=1 seems to not work
... View more
06-16-2021
01:37 AM
1 Kudo
Hello I've a mistake with my app in cluster mode with Spark. My app have 2 containers : one driver and one executor. When the executor container is killed (by an error or what else), my app doesn't give an other container or kill the current attempt to reset the job. The job is like a zombie, the driver doesn't see it and continue without any error. I missed maybe something about the yarn conf or spark-submit's params. Have you ever meet this situation ?
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN