Created 06-23-2022 12:53 PM
Hi experts,
I was wondering if it is possible to see based on the application logs alone that it is trying to find available resources from the cluster, assuming that there are currently other running jobs that are utilizing the cluster resources?
If so, how would it look like in the application logs or what do I look for?
Thanks,
Created 04-22-2024 11:17 AM
Hi @ryu ,
yes you can search the details in the RM logs with the application Id.
You will get the details of the resources asked by the application and resources provided by the RM
Created 04-23-2024 06:59 AM
If its a tez application, AM logs will show how much memory is currently allocated/consumed by the application & how much free resources available in the queue at that specific time.
eg.,
2024-04-22 23:27:20,636 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:843776, vCores:206> Free: <memory:2048, vCores:306> pendingRequests: 0 delayedContainers: 205 heartbeats: 101 lastPreemptionHeartbeat: 100
2024-04-22 23:27:30,660 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:155648, vCores:38> Free: <memory:495616, vCores:356> pendingRequests: 0 delayedContainers: 38 heartbeats: 151 lastPreemptionHeartbeat: 150
This allocation details will be logged frequently in Tez AM logs.