Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

WARN yarn.YarnAllocator: Expected to find pending requests, but found none.

WARN yarn.YarnAllocator: Expected to find pending requests, but found none.

Contributor

Hi.

 

What means :

INFO yarn.YarnAllocator: Canceling requests for 0 executor containers
WARN yarn.YarnAllocator: Expected to find pending requests, but found none.

 

I see the same info and warning alerts happening a lot from Yarn log.

 

Thanks.

1 REPLY 1

Re: WARN yarn.YarnAllocator: Expected to find pending requests, but found none.

Guru

Hi,

 

The Warning "yarn.YarnAllocator: Expected to find pending requests, but found none." is related to Dynamic Allocation of Executors. I suspect that you have set (or default value) "spark.dynamicAllocation.enabled=true". With dynamic allocation of executors, the --num-executors property is ignored. Spark dynamically determines the number of executors.

 

The warnings in this case are part of the ramping-down of executor counts. As Spark realizes that it needs fewer containers than already requested it begins to remove requests using the YarnAllocator class. The YarnAllocator class is issuing this exception to show that it thought there was going to be another container to request to deallocate, but now there is not one. It appears to be a normal part of the end of a job as executors are not needed but perhaps there is a bit of deallocation happening out of order here.

 

This warning does not affect yarn performance in anyway and there is should not be any concerns.

Don't have an account?
Coming from Hortonworks? Activate your account here