HI, we started having an OOM with the catalog .
1037386 files and directories, 752946 blocks = 1790332 total filesystem object(s).
We have plan to grow our data by many folds... hope we are not capped....
Any help would be appreciated.
As I mentioned in the impala user-list, this is probably triggered by table metadata for a specific table exceeding 1GB. This can happen for many reasons (huge number of files, blocks) but a very probable one is if a partitioned table has incremental stats. The overhead of incremental stats is 400B * #columns * #partitions and it can easily help push the metadata size of the cliff. The issue will be fixed in Impala 2.3.1. In the mean time, I would recommend droping stats and using compute stats instead of compute incremental stats. The overhead of computing full stats for large tables can be quite large, so you probably want to do it less frequently.