@Sam Hjelmfelt I don't think setting YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE in yarn-env.sh, will set the same in the ContainerLaunchContext. Have you tried setting it in the service spec itself to see if it helps ? ... View more

@john x ResourceManager will only contain the information for containers which are currently running for an application. Some containers which are already finished will not be present in ResourceManager, but they will be present in App Timeline Server. So the container list command will try to fetch from from both ResourceManager and App Timeline Server. The error shows that the App Timeline Server does not contain the application. Maybe the app timeline server had issues or was not running when this application was submitted. ... View more

@Prashant Gupta Good to know that the ResourceManager started successfully. Kindly mark the answer as accepted if the problem got resolved. ... View more

Yes. Actually a user can only run a single job at any moment. To run multiple jobs at a moment, they all need to be submit as different users. ... View more

@Soumitra Sulav The problem seems to be because the FileStatus returned by OzoneFileSystem does not have the owner field set and so its empty. As a result the ownership check fails. One workaround I see is to delete the /tmp/hadoop-yarn/staging/hdfs/.staging directory before submitting the Mapreduce job. Then this ownership check gets bypassed and the staging directory will be created again. But this means that you can't have more than one job using the /tmp/hadoop-yarn/staging/hdfs/.staging directory. So its not a good workaround, although the only available one from what I see (Apart from code change in Mapreduce/Ozone) . ... View more

Good to know you got it resolved. You can accept the answer if it helped. One more thing to note is that java debug doesn't work if more than one map container is launched in the same node. This is because both map container processes will try to listen on the debug port 8787 and might fail. ... View more

@Prasan Shetty You shouldn't move the directories under hadoop/yarn/local while the NodeManager service is running as it will affect running application containers. Stop the NodeManager service on that node, perform the move operation and then start it again ... View more

@Roberto Ayuso In spark, spark.driver.memoryOverhead is considered in calculating the total memory required for the driver. By default it is 0.10 of the driver-memory or minimum 384MB. In your case it will be 8GB * 0.1 = 9011MB ~= 9G YARN allocates memory only in increments/multiples of yarn.scheduler.minimum-allocation-mb . When yarn.scheduler.minimum-allocation-mb=4G, it can only allocate container sizes of 4G,8G,12G etc. So if something like 9G is requested it will round up to the next multiple and will allocate 12G of container size for the driver. When yarn.scheduler.minimum-allocation-mb=1G, then container sizes of 8G, 9G, 10G are possible. The nearest rounded up size of 9G will be used in this case. ... View more

@Amila Silva HDP 3.0 supports GPU isolation in docker using nvidia-docker-plugin https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin which is part of nvidia-docker v1. Currently only this is supported and not the newer version. ... View more

@Michael Bronson I will look to create an article about configuring the vcores for cpu scheduling when I get time. I will mention this part there. ... View more

Interesting. Can you paste the lscpu output of the nodes you are mentioning? ... View more

@Michael Bronson Yarn Vcores can ideally be set up to 2x the actual cpu present based on the use case. Thats why ambari provides the option in the scroll bar. It does not depend on the number of threads shown in lscpu. If you want to prevent over utilization of cpu by Yarn and leave cpu for OS and other processes you can set to 80% of 32 . But keep in mind that this value will only be considered by YARN for scheduling containers, when CPU scheduling is enabled. ... View more

@Michael Bronson The lscpu output of "CPU(s):" itself always takes into consideration the "Thread(s) per core". So usually, CPU(s) = [Thread(s) per core] X [Core(s) per socket] X [Socket(s)] It is sufficient to consider only CPU(s), when setting yarn.nodemanager.resource.cpu-vcores . ... View more

@naveen r The amount of resource a Yarn application will request is completely dependent on the type of Application. For MapReduce, it is generally based on the inputsplit/no. of reducers configured and the memory/vcores configured per mapper/reducer in the JobConf. To check how much Resources are currently being used by a single application, you can use the following Rest API and check the fields "allocatedMB","allocatedVCores" and "runningContainers" GET http://<rm http address:port>/ws/v1/cluster/apps/<applicationId> ... View more

Can click on the application in RM UI and see what is reported in Diagnostics and paste the content ? It should specify the reason why the job is still in accepted state. ... View more

An easy way to check the maximum am resource is in RM UI for queue q4 http://rm-host:8088/cluster/scheduler?openQueues=Queue:%20q4 Check the values for Max Application Master Resources and Used Application Master Resources . Also you can check other values here which will be useful to identify your queue limits configured. ... View more

There are additional parameters which limit the usage of queue resources by a single user or the application master. They are yarn.scheduler.capacity.<queue-path>.user-limit-factor and yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent . The detailed documentation of these capacity-scheduler properties can be referred at http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Properties Property Default Behaviour and Recommendation yarn.scheduler.capacity.<queue-path>.user-limit-factor 1 If you are submitting jobs as the same user, it is recommended to increase the value above 1. Otherwise the same user can't submit more than one job which exceeds the queue capacity. For q4, a single user can only utilize the max 100% if this is set to 10. This is likely the reason the new job is not getting executed. yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent 0.1 For example if you consider the default value, in q4 only 10% of the max-capacity can be used. When multiple applications are launched to the same queue, then new applications wont be accepted even if resources are free in the cluster ... View more

I didn't notice that you were only setting YARN_RESOURCEMANAGER_OPTS. This env variable is used for only the resourcemanger daemon. So to specify the opts for all hadoop and yarn client commands, you can use HADOOP_CLIENT_OPTS in . hadoop-env.sh . export HADOOP_CLIENT_OPTS="-Dyarn.resourcemanager.hostname=192.168.33.33" But I am not sure why you would need to this when you can just set it in the yarn-site.xml, which is what is recommended. ... View more

yarn-env.sh is used when you run any yarn command. So it works if you use the yarn command to submit a mapreduce job as below. yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 5 5 But spark-submit command doesn't invoke yarn-env.sh, so it will read the yarn-site.xml from $HADOOP_CONF_DIR and gets resourcemanager address from it. ... View more

The retain-seconds will not work for an active application that is writing files. It works by checking whether the last modified timestamp for the application log dir falls older than the retain-seconds. Since your streaming job writes logs continuously, the directory timestamp will never fall within 600 seconds. So your logs are not getting deleted because of this. Also log aggregation in yarn doesn't work the same way as setting log rolling/retention like in log4j as you are expecting. ... View more

The yarn log aggregation retention can be controlled by setting yarn.log-aggregation.retain-seconds property in yarn-site.xml For example, if you want logs older than 30 days to be deleted, you can set yarn.log-aggregation.retain-seconds to 2592000 ... View more

@Simran Kaur The log snippets don't indicate the problem you are facing. Can you post or attach any ERROR or failure messages in the log? ... View more