question How to predict how much memory catalogd needs? in Archives of Support Questions (Read Only)

How to predict how much memory catalogd needs?

thewayofthinkin — Fri, 16 Sep 2022 09:55:23 GMT

Hello,

I have question about memory used by catalogd.

Here is some info.

- 700 tables on Impala

- 32G physical memory on NameNode

- impala-server is not running on NameNode ( I stopped it since it uses too much memory. )

- catalogd's memory usage ( VIRT: 25.2G, RES: 16G )

My qusetion is like these.

- How to predict how much memory catalogd need?

- What factores can contribute to the memory usages of catalogd?

- Is there any way to fource catalogd to release or flush memory?

Thank you

Gatsby

Re: How to predict how much memory catalogd needs?

alex.behm — Tue, 05 Jan 2016 00:51:29 GMT

A few short responses to your questions:

- How to predict how much memory catalogd need?

The catalog caches table from the Hive Metastore as well as block location information from HDFS. The memory consumed will depende on the number of HDFS files and blocks as well as the number of databases, tables, and partitions.

- What factores can contribute to the memory usages of catalogd?

Incremental stats adds an additional memory requirement.

- Is there any way to fource catalogd to release or flush memory?

Not directly. If the catalogd is using too much memory, reducing the number of partitions and/or files/blocks should help.

Re: How to predict how much memory catalogd needs?

thewayofthinkin — Tue, 05 Jan 2016 00:53:01 GMT

Thx. 🙂