Reply
Highlighted
Contributor
Posts: 57
Registered: ‎04-26-2017

Impala Catalog JVM Size

Hi All,

I'm curious what you all have set for Java Heap Size for Impala Catalog.  At present i've had to increase to 128GB as 64GB limit was hit way to often causing OOM errors.

Total number of tables is 11,138 which are made up of 575,692 partitions

Thanks,
Chris

Contributor
Posts: 57
Registered: ‎04-26-2017

Re: Impala Catalog JVM Size

Another note is that total size of the MySQL HiveMetastore database is 102GB, so maybe need to be simialr to that.

Master
Posts: 430
Registered: ‎07-01-2015

Re: Impala Catalog JVM Size

That seemed to me too much, but I checked that for ~15k partitions I have 2.5GB, so ~150k per partition.
Contributor
Posts: 57
Registered: ‎04-26-2017

Re: Impala Catalog JVM Size

Seems about right then based upon your values.  Good to know i've not got some weird memory leak going on or something

 

Master
Posts: 430
Registered: ‎07-01-2015

Re: Impala Catalog JVM Size

I checked additionaly the mem metrics, here are the details:

memory.mapped-bytes 5.14 GB Total bytes of memory mappings in this process (the virtual memory size).
memory.rss 3.01 GB Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.
memory.total-used 153.57 MB Total memory currently used by TCMalloc and buffer pool.

So practically it has 5.14GB in my case... It's quite a lot..But as you said, it has to probably fit the entire db into a memory, because this is the purpose of this service, right?

 

Contributor
Posts: 57
Registered: ‎04-26-2017

Re: Impala Catalog JVM Size

Yes that's my understanding of what the catalog is used for.

Announcements