Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Job blocked without vcores

Highlighted

Job blocked without vcores

New Contributor

Hi,

I'm using a sandbox in my notebook. My job is stuck in 5% and never finish.

I found in the log the following line:

2016-04-06 12:02:29,936 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1459968013814_0001: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:500, vCores:0> knownNMs=1

I tried to change vcores configuration in ambari adding the following configuration in Yarn, under "Custom Yarn-site.xml":

yarn.app.mapreduce.am.resource.cpu-vcores

I tried values "1" and "2" and nothing changed.

How can I solve this problem ? Which configuration should I change in Ambari ?

Thank you!

6 REPLIES 6

Re: Job blocked without vcores

Super Guru

@Dennes Oliveira What yarn queue are you using? can you check the queue to verify you are allocated enough resources? to make it easier just assign yourself the entire sandbox bandwidth.

Re: Job blocked without vcores

New Contributor

Thank you for your help, but I don't know much about this. Can I check this in Ambari ?

Re: Job blocked without vcores

@Dennes Oliveira

What type of job are you running?

Re: Job blocked without vcores

New Contributor

Java jobs. SQL Server RC 1 Polybase is able to compile, publish and run java jobs inside the hadoop sandbox. I can see the job "arriving" in resource manager UI, but the job stops in the map phase and the last line in the log mention vcores:0, what I think is the cause for the job being blocked, but I'm not sure about how to correct this. I found this article: http://hortonworks.com/blog/managing-cpu-resources-in-your-hadoop-yarn-clusters/

But I'm not sure about where to insert these configurations, or neither if they will be enough:

map.cpu.vcores reduce.cpu.vcores yarn.app.mapreduce.am.resource.cpu-vcores I used ambari to insert these configurations under "custom yarn-site.xml", but nothing changed.

Thank you !

Re: Job blocked without vcores

Rising Star

@Dennes Oliveira check the yarn setting minimum vcores in ambari under yarn. What do you have that set to? The sandbox is only capable of running around 3 containers so that may be your issue. Are you getting a container for your application master? It sounds like you are. Also check the physical cpu limit under yarn and increase it if needed.

Re: Job blocked without vcores

New Contributor

Hi, Ian,

There were no vcores configuration in Ambari by default when I installed the sandbox. I included some configurations but I'm not sure I'm doing this in the right place or including the correct configuration.

Under "Custom Yarn-site.xml" I included the following configurations: map.cpu.vcores : 1 nodemanager.resource.cpu-vcore: 2 reduce.cpu.vcores : 1 yarn.app.mapreduce.am.resource.cpu-vcores : 2

Under Scheduler/Capacity Scheduler I included the following configuration:

yarn.scheduler.minimum-allocation-vcores=1

It seems the job is taking long to start now, but the result is the same, it stops in the map phase and reports that same line in the log, with 0 vcores.

Thank you!