There is a calculation in The Impala Cookbook to estimate the heap memory usage for metadata:   • num of tables * 5KB + num of partitions * 2KB + num of files * 750B + num of file blocks * 300B + sum(incremental col stats per table) • Incremental stats      For each table, num columns * num partitions * 400B   Usually, the insufficient catalog heap memory is caused by a large number of small files or/and partitions. For example, a single 512MB file needs (750B + 4 * 3 * 300B) = 4350B. But if we split this file to 128 4MB files, these files will use (128 * 750B + 128 * 3 * 300B) = 211200B, nearly 50 times!   ... View more

This is a bug fixed by HIVE-15642. You can work around this issue by setting hive.metastore.dml.events to false.   ... View more

The calculations on constants should be processed in the compiling phase. If you have a look at the query plans as below, you can see that both "months_add(now(), -1)" and "months_add('2020-01-01', -1)" are evaluated already. Impala doesn't need to repeat evaluation for each row. [localhost:21000] default> explain select * from mytable where tran_timestamp > months_add(now(), -1); Query: explain select * from mytable where tran_timestamp > months_add(now(), -1) +------------------------------------------------------------------------------------+ | Explain String                                                                     | +------------------------------------------------------------------------------------+ | Max Per-Host Resource Reservation: Memory=0B Threads=2                             | | Per-Host Resource Estimates: Memory=10MB                                           | | WARNING: The following tables are missing relevant table and/or column statistics. | | default.mytable                                                                    | |                                                                                    | | PLAN-ROOT SINK                                                                     | | |                                                                                  | | 01:EXCHANGE [UNPARTITIONED]                                                        | | |                                                                                  | | 00:SCAN HDFS [default.mytable]                                                     | |    HDFS partitions=1/1 files=0 size=0B                                             | |    predicates: tran_timestamp > TIMESTAMP '2021-06-30 17:08:33.432148000'          | |    row-size=16B cardinality=0                                                      | +------------------------------------------------------------------------------------+ Fetched 13 row(s) in 0.99s [localhost:21000] default> explain select * from mytable where tran_timestamp > months_add('2020-01-01', -1); Query: explain select * from mytable where tran_timestamp > months_add('2020-01-01', -1) +------------------------------------------------------------------------------------+ | Explain String                                                                     | +------------------------------------------------------------------------------------+ | Max Per-Host Resource Reservation: Memory=0B Threads=2                             | | Per-Host Resource Estimates: Memory=10MB                                           | | WARNING: The following tables are missing relevant table and/or column statistics. | | default.mytable                                                                    | |                                                                                    | | PLAN-ROOT SINK                                                                     | | |                                                                                  | | 01:EXCHANGE [UNPARTITIONED]                                                        | | |                                                                                  | | 00:SCAN HDFS [default.mytable]                                                     | |    HDFS partitions=1/1 files=0 size=0B                                             | |    predicates: tran_timestamp > TIMESTAMP '2019-12-01 00:00:00'                    | |    row-size=16B cardinality=0                                                      | +------------------------------------------------------------------------------------+ Fetched 13 row(s) in 0.02s   ... View more

If a sqoop job failed/crashed in the middle of importing a table, the table is imported. When you run this job again, it will start from zero so you need to clear the partially imported data first.  Alternatively, if you know which rows are not imported yet, you can use the WHERE clause when you restart the job to import the rest rows. ... View more

Impala shares metadata and data with Hive. You can use sqoop to import the tables from your database into hive. Don't forget to run 'invalidate metadata' in Impala after the ingestion is done. Otherwise, you can't see the imported tables. ... View more