Created on 02-09-2023 05:59 AM - edited 02-09-2023 06:02 AM
When I submit a job through yarn rest api, it is initiating two applications by default .One application from yarn user (Dr.who) which will stay in Accepted state and fail after sometime . Another application from local which is running successfully. There is few seconds time difference between two applications, and both the app id's are different. Even though there are sufficient resources AM of the yarn rest api submitted job is unable use the resources and initiating job from local where AM of local job is using resources and running job successfully.
Created 02-26-2023 05:08 AM
I see a disabled log aggregation for these applications. You can enable the log aggregation and look for possible reasons in the application logs.
To view the entire log for an application
yarn logs -applicationId <application-id>
Below is a sample snippet of yarn-site.xml to enable log aggregation.
You can look up or add these properties in the Hadoop management application (like cloudera-manager) as well.
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>3600</value>
</property>