When I submit a job through yarn rest api,  it is initiating two applications by default .One application from yarn user (Dr.who) which will stay in Accepted state and  fail after sometime . Another application from local which  is running successfully. There is few seconds time difference between two applications, and both the app id's are different. Even though there are sufficient resources AM of the yarn rest api submitted job is unable use the resources and initiating job from local where AM of local job is using resources and running job successfully. ... View more

I am submitting job through yarn rest api, and it is creating duplicate job with another application id,after a 5-10 seconds. The job which I have submitted through yarn rest api is being failed and the duplicate job is being succeeded. LOGS: yarn submitted application looks like : Application Node Label expression: <DEFAULT_PARTITION> AM container Node Label expression:<DEFAULT_PARTITION> Duplicate job application looks like: Application Node Label expression:<Not set> AM container Node Label expression:<DEFAULT_PARTITION>   I tried with changing the configurations for the application node label expression by adding the following property in the yarn-site.xml: <property> <name>yarn.node-labels.am.default-node-label-expression</name> <value>Not set</value> </property> and tried to specify the application node label expression through the rest api. But no use of it ,I am thinking that if I am able to change the yarn application node label expression to <Not set> problem will be solved.If any knows or experienced this before please help me. Thanks in advance. ... View more

When I submitted a job through Yarn rest api , It is being in accepted state which is with user dr.who and after 5-10 seconds a new job with new application Id is initiating with my username and  the new job is being succeeded. dr.who job is being failed I am new to hadoop can anyone experienced this issue before suggest me to come out of this.  ... View more

I am able to submit job through the terminal and if I am using the same command and submit the job through the yarn rest api it not submitting and one more thing is even I specified the queue as 'a'  but it is taking queue as default. As I am just beginner I am not able encounter the problem can anyone help getting out of this please . THANKS IN ADVANCE       { 'application-id': 'application_1675072653900_0123', 'application-name': None, 'am-container-spec': { 'commands': { 'command': '/home/kirankathe/Desktop/spark-3.3.1-bin-hadoop3/bin/spark-submit --master yarn --deploy-mode client --packages org.mongodb.spark:mongo-spark-connector_2.12:3.0.1 --driver-memory 1g --executor-memory 1g --num-executors 1 --executor-cores 2 --queue a --class org.apache.spark.examples.SparkPi hdfs://localhost:9000/user/kirankathe/word_count.py' }, 'local-resources': { 'entry': [{ 'key': 'word_count.py', 'value': { 'resource': 'hdfs://localhost:9000/user/kirankathe/word_count.py', 'type': 'FILE', 'visibility': 'APPLICATION', 'timestamp': 1675259542629 } }] } }, 'application-type': 'SPARK', 'priority': 0, 'max-app-attempts': 1 } ... View more