I'm using a sandbox in my notebook. My job is stuck in 5% and never finish.
I found in the log the following line:
2016-04-06 12:02:29,936 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1459968013814_0001: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:500, vCores:0> knownNMs=1
I tried to change vcores configuration in ambari adding the following configuration in Yarn, under "Custom Yarn-site.xml":
I tried values "1" and "2" and nothing changed.
How can I solve this problem ? Which configuration should I change in Ambari ?
Java jobs. SQL Server RC 1 Polybase is able to compile, publish and run java jobs inside the hadoop sandbox. I can see the job "arriving" in resource manager UI, but the job stops in the map phase and the last line in the log mention vcores:0, what I think is the cause for the job being blocked, but I'm not sure about how to correct this. I found this article: http://hortonworks.com/blog/managing-cpu-resources-in-your-hadoop-yarn-clusters/
But I'm not sure about where to insert these configurations, or neither if they will be enough:
map.cpu.vcores reduce.cpu.vcores yarn.app.mapreduce.am.resource.cpu-vcores I used ambari to insert these configurations under "custom yarn-site.xml", but nothing changed.
Thank you !
@Dennes Oliveira check the yarn setting minimum vcores in ambari under yarn. What do you have that set to? The sandbox is only capable of running around 3 containers so that may be your issue. Are you getting a container for your application master? It sounds like you are. Also check the physical cpu limit under yarn and increase it if needed.
There were no vcores configuration in Ambari by default when I installed the sandbox. I included some configurations but I'm not sure I'm doing this in the right place or including the correct configuration.
Under "Custom Yarn-site.xml" I included the following configurations: map.cpu.vcores : 1 nodemanager.resource.cpu-vcore: 2 reduce.cpu.vcores : 1 yarn.app.mapreduce.am.resource.cpu-vcores : 2
Under Scheduler/Capacity Scheduler I included the following configuration:
It seems the job is taking long to start now, but the result is the same, it stops in the map phase and reports that same line in the log, with 0 vcores.