Reply
Explorer
Posts: 18
Registered: ‎04-26-2017

Impala using SWAP despite Kill When Out of Memory set to active

Hi,

I have various hosts which start to use swap and this happenes when impala goes beyond it's mem_limit.

At present mem_limit is set to 128GB per data node, however as the charts below show it spiked and went above this which then trigged the swap usage (which does not get released).

My question is why is impala using more memory than allowed and going to swap when I have set kill out of memory to be activ?  Swapiness is set to 1.

Impala memory.PNG

Cloudera Employee
Posts: 25
Registered: ‎12-11-2015

Re: Impala using SWAP despite Kill When Out of Memory set to active

Hi,

 

The decision of whether to use swap or not is done by the kernel. Impala will not have a control over it. If incase impalad running on a node reaches near the mem_limit set, then the query will error out with memory limit exceeded error.

 

The below link explains when swapping occurs

 

http://careers.directi.com/display/tu/Understanding+and+optimizing+Memory+utilization

Quote from the link:
==============
Swapping occurs in one of two scenarios -
* When the kernel needs to allocate a page of memory to a process and finds that there is no memory available. In this case the kernel must swap out the least used pages of an existing process into the swap space (on disk) and allocate those page frames to the requesting process
* There is a kernel parameter that determines swappiness of the kernel. The value is between 1 to 100 and is set to around 60 by default. A value of 100 means that the kernel will be considerably agressive when it comes to preferring allocatoin of memory to disk cache over processes. A value of 60 can result in occasional swapping out of process owned pages onto disk to make room for additional pages for the disk cache

 

--

 

These are indications of the node's memory being fully utilised and the subsequent step would be to investigate on 1. if the node being overcommitted with multiple roles, then offload some roles to other nodes

2. If you notice the node swapping to disk even if there is enough free RAM, then it would be best to check if NUMA is disabled - Here is a link which describes such scenarios https://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ 

3. If the cluster is suffering from overload, then its best to think about scaling out.

Champion
Posts: 564
Registered: ‎05-16-2016

Re: Impala using SWAP despite Kill When Out of Memory set to active

To add  points to @venkatsambath 

 

Cloduera recommends to you use vm.swapiness=10. when it comes to RHEL 6.4 or any higer version of kernel

for better performance.

vi  /proc/sys/vm/swappiness

https://www.cloudera.com/documentation/enterprise/5-3-x/topics/cdh_admin_performance.html

Explorer
Posts: 18
Registered: ‎04-26-2017

Re: Impala using SWAP despite Kill When Out of Memory set to active

Thanks for the response, its a great help.

One thing I still don't understand if why does the impala daemon resident memort chart I attached, show that the memory goes above that of impala memory limit?

Explorer
Posts: 18
Registered: ‎04-26-2017

Re: Impala using SWAP despite Kill When Out of Memory set to active

Thanks for the response, its a great help.

One thing I still don't understand if why does the impala daemon resident memort chart I attached, show that the memory goes above that of impala memory limit?

Cloudera Employee
Posts: 266
Registered: ‎07-29-2015

Re: Impala using SWAP despite Kill When Out of Memory set to active

@chriswalton007Not sure if this is what you're seeing, but the Impala daemon memory limit does not include the embedded JVM: https://issues.apache.org/jira/browse/IMPALA-691

Highlighted
Explorer
Posts: 18
Registered: ‎04-26-2017

Re: Impala using SWAP despite Kill When Out of Memory set to active

Thanks Tim, its possible that is the case.

Am I correct in thinking there is no option to set the JVM memory size limit for impala?  I can't seem to find it within Cloudera?

In the bug link you provided it refers to FE (JVM).  Is FE an acroymn for something?

Announcements