How to predict how much memory catalogd needs?

[#1]

Hello,

I have question about memory used by catalogd.

Here is some info.

- 700 tables on Impala

- 32G physical memory on NameNode

- impala-server is not running on NameNode ( I stopped it since it uses too much memory. )

- catalogd's memory usage ( VIRT: 25.2G, RES: 16G )

My qusetion is like these.

- How to predict how much memory catalogd need?

- What factores can contribute to the memory usages of catalogd?

- Is there any way to fource catalogd to release or flush memory?

Thank you

Gatsby

[#2 · Accepted solution]

A few short responses to your questions:

- How to predict how much memory catalogd need?

The catalog caches table from the Hive Metastore as well as block location information from HDFS. The memory consumed will depende on the number of HDFS files and blocks as well as the number of databases, tables, and partitions.

- What factores can contribute to the memory usages of catalogd?

Incremental stats adds an additional memory requirement.

- Is there any way to fource catalogd to release or flush memory?

Not directly. If the catalogd is using too much memory, reducing the number of partitions and/or files/blocks should help.

[#3]

Thx. 🙂