Hello everyone.   I am trying to train a ML model on a cluster by using a docker image and spark-submit with yarn. I already tried to follow this process before on a training cluster I made and I succeeded.   But when I run it this time, yarn prompts that one of the mounts is invalid. Of course I tried with Kerborise and without, both didn't work and both runs didn't imply any problems related to connection so we are good from this side.     This is what I tried: spark-submit \ --master yarn \ --deploy-mode cluster \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=image:v1 \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data01/yarn/nm/:/data01/yarn/nm/:ro,/data02/yarn/nm/:/data02/yarn/nm/:ro,/data03/yarn/nm/:/data03/yarn/nm/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=image:v1 \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data01/yarn/nm/:/data01/yarn/nm/:ro,/data02/yarn/nm/:/data02/yarn/nm/:ro,/data03/yarn/nm/:/data03/yarn/nm/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \ modeling.py     And this is the results I got:   2022-08-10 17:43:17,694 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1658823376901_2680 failed 2 times due to AM Container for appattempt_1658823376901_2680_000002 exited with exitCode: -1 Failing this attempt.Diagnostics: [2022-08-10 17:43:17.686]Exception from container-launch. Container id: container_e43_1658823376901_2680_02_000001 Exit code: -1 Exception message: Invalid mount : /data03/yarn/nm Shell error output: <unknown> Shell output: <unknown> [2022-08-10 17:43:17.687]Container exited with a non-zero exit code -1. [2022-08-10 17:43:17.687]Container exited with a non-zero exit code -1.     P.S: I followed all the instructions and documentations needed to run this. And made all the necessary configs. Like I said before I already ran this on another cluster.   Any help would be greatly appreciated.   ... View more

Hello all,   I am trying to run spark job using spark-submit with a docker image over yarn.    I followed the instructions in the Blog provided by cloudera in the following link: https://blog.cloudera.com/introducing-apache-spark-on-docker-on-top-of-apache-yarn-with-cdp-datacenter-release/   and I ran into an error that I couldn't fine an answer to. Note: I already did all the configurations required from the post.   I ran this command : spark-submit \ --master yarn \ --deploy-mode cluster \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \ --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \ --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \ ols.py     And this is the error I get:     Sometimes its gives me an exit code 29. I don't understand what the problem is especially that I followed instructions properly.       ... View more