Reply
Highlighted
Contributor
Posts: 39
Registered: ‎10-29-2015

Swap Memory Usage for HDFS, Yarn and Impala

Hello,

 

We have been getting regular warning messages of Swap Memory being utilized more than the threshold limit.

 

Currently, we have set swap memory threshold as below:

For,

HDFS = 100 MB

Impala = 30 MB

Yarn = 500 MB (Could be +-20%, I dont remember the exact number).

 

Swap Memory usage for each of the component is crossing the threshold and reaches upto 720 MB (in case of Yarn). Because of this, we usually see warnings on our CM dashboard.

 

I know increasing the Swap Memory Threshold could remove these warnings, however, we would prefer rather reducing the usage.

 

Would be great if anyone could suggestion any Memory Tuning options which would reduce the usage of Swap Memory. Also, if there is a best recommendations of setting memory usage threshold, kindly share that as well.

 

Thanks

Snm

New Contributor
Posts: 6
Registered: ‎09-21-2017

Re: Swap Memory Usage for HDFS, Yarn and Impala

Hi,

 

did you set the swappiness in the OS?

 

Best

Andy

Posts: 1,108
Topics: 1
Kudos: 285
Solutions: 134
Registered: ‎04-22-2014

Re: Swap Memory Usage for HDFS, Yarn and Impala

@snm1523,

 

Cloudera recommends the following:

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_performance.html#cdh_perfo...

 

Essentially, we don't want swapping to occur.

Contributor
Posts: 39
Registered: ‎10-29-2015

Re: Swap Memory Usage for HDFS, Yarn and Impala

Hello Vogone,

Thanks for the reply.

Yes, swappiness is set to 1.

Regards,
snm
Contributor
Posts: 39
Registered: ‎10-29-2015

Re: Swap Memory Usage for HDFS, Yarn and Impala

Hello @bgoogley,

Thank you for the reply.

We have already set the swappiness to 1. Still getting these swappiness warnings.

Regards,
Snm
New Contributor
Posts: 2
Registered: ‎04-10-2018

Re: Swap Memory Usage for HDFS, Yarn and Impala

@snm1523,

 

Disk IO intensive workloads can cause the kernel to swap even if swappiness is set to 1.

 

Have you tried increasing the vm.vfs_cache_pressure kernel parameter on all worker nodes? Here is a post on how to fine tune the parameter: http://datavelo.com/en/2018/04/10/kernel-swapping-vm-swappiness-1

 

Best,

z

Contributor
Posts: 36
Registered: ‎07-20-2016

Re: Swap Memory Usage for HDFS, Yarn and Impala

Any working solution for the swap problem?

 

We have same problem on CentOS 7 with CDH Enterprise 5.14.0

 

DataNode, NodeManager, YarnChild processes are swapping