Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Catalog JVM Size

Impala Catalog JVM Size

Rising Star

Hi All,

I'm curious what you all have set for Java Heap Size for Impala Catalog.  At present i've had to increase to 128GB as 64GB limit was hit way to often causing OOM errors.

Total number of tables is 11,138 which are made up of 575,692 partitions

Thanks,
Chris

5 REPLIES 5

Re: Impala Catalog JVM Size

Rising Star

Another note is that total size of the MySQL HiveMetastore database is 102GB, so maybe need to be simialr to that.

Re: Impala Catalog JVM Size

Master Collaborator
That seemed to me too much, but I checked that for ~15k partitions I have 2.5GB, so ~150k per partition.

Re: Impala Catalog JVM Size

Rising Star

Seems about right then based upon your values.  Good to know i've not got some weird memory leak going on or something

 

Re: Impala Catalog JVM Size

Master Collaborator

I checked additionaly the mem metrics, here are the details:

memory.mapped-bytes 5.14 GB Total bytes of memory mappings in this process (the virtual memory size).
memory.rss 3.01 GB Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.
memory.total-used 153.57 MB Total memory currently used by TCMalloc and buffer pool.

So practically it has 5.14GB in my case... It's quite a lot..But as you said, it has to probably fit the entire db into a memory, because this is the purpose of this service, right?

 

Re: Impala Catalog JVM Size

Rising Star

Yes that's my understanding of what the catalog is used for.