11-27-2017 05:09 AM - last edited on 11-27-2017 06:05 AM by cjervis
We have a CHD 5.8.3 cluster, each node has hdfs, yarn and Impalad processes.
We saw that we have red alerts on the Impala service on swap issues,
We did all the calculation with OS, YARN and HDFS process memory allocation per server so there will not be memory over allocation.
After it, we've found that Impala has JVM heap memory except for mem_limit so we set the max heap to 8G so it will be controlled (and remove 8G from the other components on the node).
But now we still see swap issues on Impala and other Hadoop services and our consumers complain about slowness and out of memory on Impala queries
Can you help us to figure why we have red alerts on Impala service in the Cloudera manager? What other reason can be for this?
11-27-2017 05:35 AM
11-27-2017 06:31 AM
Right now we're on swappiness=1,
We don't want to set it to 0 and risk getting OOM so we set it low as possible.
We save in each server 8GB RAM for the OS and 1GB for HDFS and Yarn and Cloudera agent processes,
I guess it's enough right?
Another point besides this,
we have NUMA of 2 groups on each of our nodes, what is the best practice of Hadoop (Hive, Impala etc) about it? disable/enable it?
12-03-2017 06:42 AM
anyone can shed a light on NUMA and Hadoop, should we disable/keep enabled?
Why is Impala process going to swap even though we defined mem_limit and max heap for its JVM?
Thanks for your help!
12-04-2017 09:12 AM
I am not sure of Impala's stance on NUMA as it is out of my specialty area. I can comment some on your other points, however. Impala does not swap, the OS will swap out memory when needed as demands require if you have swappiness set to 1. In Cloudera Manager, you can view a chart of the host's memory usage. Looking more closely at information regarding the memory pressure and I/O use on the OS (caused by all processes, not just Impala) is a better way to guage what is happening here. Swapping is a function of the kernel, so it is not aware of "Impala". If you have swappiness set to "1" and you are still seeing swapping, that indicates the OS had demands put on it that required it to use swap to prevent OOM conditions.
So, in short, you are still seeing swapping since the OS deemed it necessary. Looking more closely about the overall resource usage on the host or hosts in question is a good start.
12-05-2017 12:18 AM
Hi @bgooley and all,
We've put a script that reports every 2/3 seconds on the top memory process on the server and the OS memory and swap status.
From what I've seen there is still free and cached space in the OS when the OS is using swap,
it mostly happens when Impala memory is growing.
Maybe it's because Impala needs large continuous segment in the memory and can't have one or maybe Impala wants more memory than memory_limit + its JVM max heap size?
Is there something else I should notice in Hadoop (or more specific Impala) that needs memory and I've missed?
Are there parameters I should tweak and may help and why it helps?
Our users complain about slowness and out of memory errors on Impala queries they running this can be related to this? cause Cloudera showing the alerts of swap in red color so it seems important.
Thanks for your help,
04-10-2018 03:11 AM
Have you tried increasing the vm.vfs_cache_pressure kernel parameter on all worker nodes?
Here is a summary on my experience with this parameter: http://datavelo.com/en/2018/04/10/kernel-swapping-vm-swappiness-1
04-12-2018 12:24 AM