In the capacity scheduler your could set a high priority queue at 90% of the cluster with extension( max. Capacity) to 100% and a low priority queue wirh 10% with extension( max capacity) to 100%. In this case jobs the first queue would always get 90% of the cluster if it needs it and the second queue would only get a tiny amount of the cluster if the high priority Queues have Queries. The low priority queue would still be able to monopolize the cluster if if has very long running tasks. But you could fix that with preemption. ( Or by making sure tasks in your cluster don't run for too long which they shouldn't anyway.) ... View more

Hajime, the above scripts are for the yarn container and mapreduce memory settings. If you are trying to configure the memory of the nodemanager process itself then that shouldn't need more than 2GB - 4GB. If you are seeing outOfMemory there I suggest you turn on verbose GC for the nodemanager process and review the GC logs. ... View more

@Hajime This makes sense hive.exec.reducers.bytes.per.reducer Default Value: 1,000,000,000 prior to Hive 0.14.0; 256 MB ( 256,000,000 ) in Hive 0.14.0 and later Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917) Size per reducer. The default in Hive 0.14.0 and earlier is 1 GB, that is, if the input size is 10 GB then 10 reducers will be used. In Hive 0.14.0 and later the default is 256 MB, that is, if the input size is 1 GB then 4 reducers will be used. Point to note: Calculate hive.exec.reducers.max should be set to a number which is less than the available reduce slots on the cluster. Hive calculate the reducers based on hive.exec.reducers.bytes.per.reducer (default 1GB). Consider setting this high based on the workloads and demand for the reducers on the cluster https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties ... View more