Free the impala resident memory without restarting the impala service

[Fawze]

Hi Guys,

I have a cluster with 54 nodes, i configured the 35 GB for the impala daemon, but all the time there is ~ 10GB memory that each nodes reserved it as resident memory.

Is there a way to free up this memory without restarting the impala service? is it a configuration parameter that i can used to disable this?

My impala version is 2.3 and plannning to upgrade to 2.10 in the next month.

[tianyiwang]

Currently there isn't a way to free up memory. Using less nodes as coodinators may help because Impala coordinators consume more memory than non-coodinator nodes.

[Fawze]

This will not help ... any node who will exceed the memory limit will
result in the query fail ....

Where this resident memory can be used? Should it occupy part of the impala
daemon memory limit, if node has impala daemon memory limit of 32G and the
residen memory of impala is 10G, is it mean that I have only 22G to be used
by the queries on this node?

[Tim Armstrong]

One thing to keep in mind is that resident memory includes the heap of the embedded JVM, which is not included in the Impala daemon's mem_limit for legacy reasons. 

Impala daemon memory consumption should be limited (approximately) by <JVM memory> + <mem_limit>. Within mem_limit there will be some overhead for system memory - up to a few GB, depending on various factors. 

There are various memory breakdowns on this debug page that show the size of the JVM heap, amongst other things http://impalad-host:25000/memz .

If you look at the line below, "Limit - Total" is the remaining memory that is available for queries.

Process: Limit=120.00 GB Total=1341.57 MB Peak=22385.06 MB

[Fawze]

What could let a node to resident 20G with no running queries ...

That's mean in my cluster i have 1 TB memory resident.

[Tim Armstrong]

@Fawzecould be JVM memory - you can check the JVM heap size on /memz debug page as I mentioned in a previous comment.

[Fawze]

Memory consumption / limit: 2.79 GB / 35.00 GB

tcmalloc

------------------------------------------------
MALLOC:     2767220936 ( 2639.0 MiB) Bytes in use by application
MALLOC: +     40321024 (   38.5 MiB) Bytes in page heap freelist
MALLOC: +     72362968 (   69.0 MiB) Bytes in central cache freelist
MALLOC: +        38144 (    0.0 MiB) Bytes in transfer cache freelist
MALLOC: +     98283104 (   93.7 MiB) Bytes in thread cache freelists
MALLOC: +     39088280 (   37.3 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =   3017314456 ( 2877.5 MiB) Actual memory used (physical + swap)
MALLOC: +  16260947968 (15507.6 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  19278262424 (18385.2 MiB) Virtual address space used
MALLOC:
MALLOC:          26327              Spans in use
MALLOC:            454              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

I can see:  

MALLOC: +  16260947968 (15507.6 MiB) Bytes released to OS (aka unmapped

which is 15 GB. 

@Tim Armstrong Can you please also advise where u can use this and when:

Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

[Tim Armstrong]

@Fawzethat memory is already released to the OS and so won't count against resident memory. I.e. that memory doesn't matter.

I forgot that /memz doesn't have the JVM metrics in the 2.3 release. You should be able to see the JVM usage on the /metrics page in that version, e.g. the "jvm.total.committed-usage-bytes" metric.

Another case we've seen is if transparent huge pages is set to "always", the resident memory can bloat. You can check the setting on your host by looking at one of the following files:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

If it's set to "always", I'd recommend switching it to "madvise", or "never" if madvise is not available. See https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_performance.html#cdh_perfo... for details on how to change it permanently.

[Fawze]

[root@svpr-dhc013 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@svpr-dhc013 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]

@Tim Armstrongthat memory is already released to the OS and so won't count against resident memory. I.e. that memory doesn't matter.

what do you mean by this, it will not count on the memory_limit for the query? or it will not count as resident memory?  what can cause impala jvm to reserve such huge memory, is this memory can be used by yarn or impala, i assume no, so i think this memory is matter, what do you think?

Can you help me also to understand this:

Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

[Tim Armstrong]

The "memory released by the OS" is virtual memory regions that aren't backed by any physical memory. They are not resident memory and aren't counted against queries.

That message about "ReleaseFreeMemory()" is exposed from a library that Impala uses (TCMalloc) and isn't actionable for an Impala user or administrator.

The JVM memory is distinct from any of the above memory and does count as resident memory.