One common misconception with YARN is that it 'preallocates' resources. It does not. The memory requests and the CPU requests are not 'limited' in any pre-reserved manner. For memory checks, the NodeManagers run a simple monitor that periodically checks if the container (child process) it spawned is exceeding the request granted to it. If the pmem usage is higher, it is enforced with a kill sent to the process. The same is not done for other resources yet - only memory. You're correct that CGroups can help in enforcing CPUs/etc. See https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_service_pools.html > However despite that configuration one can see more than 20 threads open for every container, whereas I'd expect no more than 2 The thread count depends on what the code runs. For ex. HDFS client work typically runs up 3-4 threads by itself, because it is architected that way. However, this does not indicate load by itself - that is driven by how the code uses the HDFS client. Thread count alone is not a good indicator, but active thread count over time may be a better metric. > Does anyone know why a single container uses so much CPU time? Difficult to say. For straight-forward record-by-record work (such as map tasks) I'd expect ~100% CPU most of the task's life, except parts that benefit from parallelism (sort, etc.) within the framework. Outside of this, the user code is also typically free to run however it wants in the container JVM, and it may decide to run some work concurrently depending on what its goals are (once the data is read, or when writing the data). > Is it possible to control number of threads per container with some MapRed configuration or java settings? Some of the thread counts such as a Reducer's parallel fetcher thread pool can be configured. Others (such as the Responder/Reader/Caller thread arch. of a HDFS client) are not configurable since they are designed to work that way. > Or maybe this problem comes from flaws in the MapRed program itself? This could be very possible if its only affecting a partial set of your overall container workload. A jstack can help you see which threads are working on code from your own organization vs. the framework itself. > Would cgroups be a good approach to solve this issue? Yes, Linux can help 'enforce' the CPU shares. Following https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_service_pools.html may help with that. However, its worth negotiating with the developers of the jobs on requesting higher vcore counts for their CPU-heavy work before you take this step, as a visibly large slowdown may not be very favorable.
... View more
