Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can we see if the application logs is having resource issues?

avatar
Contributor

Hi experts,

I was wondering if it is possible to see based on the application logs alone that it is trying to find available resources from the cluster, assuming that there are currently other running jobs that are utilizing the cluster resources?

 

If so, how would it look like in the application logs or what do I look for?

 

Thanks,

2 REPLIES 2

avatar
Expert Contributor

Hi @ryu ,

yes you can search the details in the RM logs with the application Id.

You will get the details of the resources asked by the application and resources provided by the RM

avatar
Expert Contributor

If its a tez application, AM logs will show how much memory is currently allocated/consumed by the application & how much free resources available in the queue at that specific time.

eg.,

2024-04-22 23:27:20,636 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:843776, vCores:206> Free: <memory:2048, vCores:306> pendingRequests: 0 delayedContainers: 205 heartbeats: 101 lastPreemptionHeartbeat: 100

2024-04-22 23:27:30,660 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:155648, vCores:38> Free: <memory:495616, vCores:356> pendingRequests: 0 delayedContainers: 38 heartbeats: 151 lastPreemptionHeartbeat: 150

This allocation details will be logged frequently in Tez AM logs.