Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to predict how much memory catalogd needs?

Solved Go to solution

How to predict how much memory catalogd needs?

Expert Contributor

Hello,

 

I have question about memory used by catalogd.

 

Here is some info.

- 700 tables on Impala

- 32G physical memory on NameNode

- impala-server is not running on NameNode ( I stopped it since it uses too much memory. )

- catalogd's memory usage ( VIRT: 25.2G, RES: 16G )

 

My qusetion is like these.

- How to predict how much memory catalogd need?

- What factores can contribute to the memory usages of catalogd?

- Is there any way to fource catalogd to release or flush memory?

 

Thank you

Gatsby

 

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to predict how much memory catalogd needs?

Master Collaborator

A few short responses to your questions:

 

- How to predict how much memory catalogd need?

The catalog caches table from the Hive Metastore as well as block location information from HDFS. The memory consumed will depende on the number of HDFS files and blocks as well as the number of databases, tables, and partitions.

 

- What factores can contribute to the memory usages of catalogd?

 

Incremental stats adds an additional memory requirement.

 

- Is there any way to fource catalogd to release or flush memory?

Not directly. If the catalogd is using too much memory, reducing the number of partitions and/or files/blocks should help.

 

 

2 REPLIES 2

Re: How to predict how much memory catalogd needs?

Master Collaborator

A few short responses to your questions:

 

- How to predict how much memory catalogd need?

The catalog caches table from the Hive Metastore as well as block location information from HDFS. The memory consumed will depende on the number of HDFS files and blocks as well as the number of databases, tables, and partitions.

 

- What factores can contribute to the memory usages of catalogd?

 

Incremental stats adds an additional memory requirement.

 

- Is there any way to fource catalogd to release or flush memory?

Not directly. If the catalogd is using too much memory, reducing the number of partitions and/or files/blocks should help.

 

 

Highlighted

Re: How to predict how much memory catalogd needs?

Expert Contributor
Thx. :)
Don't have an account?
Coming from Hortonworks? Activate your account here