11-25-2017 08:21 AM
Hi Guys,
I have a cluster with 54 nodes, i configured the 35 GB for the impala daemon, but all the time there is ~ 10GB memory that each nodes reserved it as resident memory.
Is there a way to free up this memory without restarting the impala service? is it a configuration parameter that i can used to disable this?
My impala version is 2.3 and plannning to upgrade to 2.10 in the next month.
11-27-2017 04:46 PM
Currently there isn't a way to free up memory. Using less nodes as coodinators may help because Impala coordinators consume more memory than non-coodinator nodes.
11-27-2017 09:11 PM
11-28-2017 10:07 AM
One thing to keep in mind is that resident memory includes the heap of the embedded JVM, which is not included in the Impala daemon's mem_limit for legacy reasons.
Impala daemon memory consumption should be limited (approximately) by <JVM memory> + <mem_limit>. Within mem_limit there will be some overhead for system memory - up to a few GB, depending on various factors.
There are various memory breakdowns on this debug page that show the size of the JVM heap, amongst other things http://impalad-host:25000/memz .
If you look at the line below, "Limit - Total" is the remaining memory that is available for queries.
Process: Limit=120.00 GB Total=1341.57 MB Peak=22385.06 MB
12-08-2017 08:04 AM
12-08-2017 11:50 AM
@Fawzecould be JVM memory - you can check the JVM heap size on /memz debug page as I mentioned in a previous comment.
12-08-2017 12:38 PM - edited 12-08-2017 12:40 PM
Memory consumption / limit: 2.79 GB / 35.00 GB
tcmalloc
------------------------------------------------ MALLOC: 2767220936 ( 2639.0 MiB) Bytes in use by application MALLOC: + 40321024 ( 38.5 MiB) Bytes in page heap freelist MALLOC: + 72362968 ( 69.0 MiB) Bytes in central cache freelist MALLOC: + 38144 ( 0.0 MiB) Bytes in transfer cache freelist MALLOC: + 98283104 ( 93.7 MiB) Bytes in thread cache freelists MALLOC: + 39088280 ( 37.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3017314456 ( 2877.5 MiB) Actual memory used (physical + swap) MALLOC: + 16260947968 (15507.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 19278262424 (18385.2 MiB) Virtual address space used MALLOC: MALLOC: 26327 Spans in use MALLOC: 454 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.
I can see:
MALLOC: + 16260947968 (15507.6 MiB) Bytes released to OS (aka unmapped
which is 15 GB.
@Tim Armstrong Can you please also advise where u can use this and when:
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.
12-08-2017 03:15 PM
@Fawzethat memory is already released to the OS and so won't count against resident memory. I.e. that memory doesn't matter.
I forgot that /memz doesn't have the JVM metrics in the 2.3 release. You should be able to see the JVM usage on the /metrics page in that version, e.g. the "jvm.total.committed-usage-bytes" metric.
Another case we've seen is if transparent huge pages is set to "always", the resident memory can bloat. You can check the setting on your host by looking at one of the following files:
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
If it's set to "always", I'd recommend switching it to "madvise", or "never" if madvise is not available. See https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_performance.html#cdh_perfo... for details on how to change it permanently.
12-08-2017 05:07 PM - edited 12-08-2017 05:09 PM
[root@svpr-dhc013 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@svpr-dhc013 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]
@Tim Armstrongthat memory is already released to the OS and so won't count against resident memory. I.e. that memory doesn't matter.
what do you mean by this, it will not count on the memory_limit for the query? or it will not count as resident memory? what can cause impala jvm to reserve such huge memory, is this memory can be used by yarn or impala, i assume no, so i think this memory is matter, what do you think?
Can you help me also to understand this:
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
12-11-2017 11:15 AM
The "memory released by the OS" is virtual memory regions that aren't backed by any physical memory. They are not resident memory and aren't counted against queries.
That message about "ReleaseFreeMemory()" is exposed from a library that Impala uses (TCMalloc) and isn't actionable for an Impala user or administrator.
The JVM memory is distinct from any of the above memory and does count as resident memory.