Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

AM of the yarn rest api submitted job is unable to use the resources leading to failure of the application

avatar
Explorer

When I submit a job through yarn rest api,  it is initiating two applications by default .One application from yarn user (Dr.who) which will stay in Accepted state and  fail after sometime . Another application from local which  is running successfully. There is few seconds time difference between two applications, and both the app id's are different. Even though there are sufficient resources AM of the yarn rest api submitted job is unable use the resources and initiating job from local where AM of local job is using resources and running job successfully.Screenshot from 2023-02-09 18-11-41.pngScreenshot from 2023-02-09 18-12-05.pngScreenshot from 2023-02-09 18-12-13.png

1 REPLY 1

avatar
Rising Star

I see a disabled log aggregation for these applications. You can enable the log aggregation and look for possible reasons in the application logs.

To view the entire log for an application
yarn logs -applicationId <application-id>

Below is a sample snippet of yarn-site.xml to enable log aggregation.
You can look up or add these properties in the Hadoop management application (like cloudera-manager) as well.

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>3600</value>
</property>