Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

catalog-server.cc:279] OutOfMemoryError: Requested array size exceeds VM limit

catalog-server.cc:279] OutOfMemoryError: Requested array size exceeds VM limit

Explorer

HI, we started having an OOM with the catalog .

 
E1203 15:00:07.379981 24194 catalog-server.cc:279] OutOfMemoryError: Requested array size exceeds VM limit
 
We get this every 4 sec.
WE restarted catalog and it works fine for a while and then the error start reoccurring.
 
Our biggest table has 5k partitions and less than 20k files.
Catalogd is using between 15-18GB of memory and we are on impala 2.2 .
 
Have we hit a hard limit here? 
 
From the name node:

1037386 files and directories, 752946 blocks = 1790332 total filesystem object(s).

 

We have plan to grow our data by many folds... hope we are not capped....

Any help would be appreciated.

 

 
 
 

 

1 REPLY 1
Highlighted

Re: catalog-server.cc:279] OutOfMemoryError: Requested array size exceeds VM limit

Contributor

As I mentioned in the impala user-list, this is probably triggered by table metadata for a specific table exceeding 1GB. This can happen for many reasons (huge number of files, blocks) but a very probable one is if a partitioned table has incremental stats. The overhead of incremental stats is 400B * #columns * #partitions and it can easily help push the metadata size of the cliff. The issue will be fixed in Impala 2.3.1. In the mean time, I would recommend droping stats and using compute stats instead of compute incremental stats. The overhead of computing full stats for large tables can be quite large, so you probably want to do it less frequently. 

 

Dimitris