I've attached the OOM heapdump analysis. Looks like the memory it is being used by the ORC writers, but would appreciate other opinions. There also seems to be a lot of memory used by org.apache.hadoop.hive.serde2.io.HiveDecimalWritable which has 96 million objects? I'm not a Java expert so this is guesswork on my part right now. I wonder if we can use some of these figures to reverse engineer a formula? Thanks, Steve heapdump-system-overview.zip heapdump-top-components.zip ... View more

Thanks for the reply. I realise my original post is short on specific details which may help, so here goes. This is a single node test system whose configuration is as follows: llap_headroom_space = 24576<br>hive.llap.daemon,yarn.container.mb = 11264<br>llap_heap_size = 8192<br>hive.llap.io.memory.size = 2048<br>num_llap_nodes_for_llap_daemons = 1<br>hive.llap.daemon.num.executors = 1 The table being inserted into is an ORC table with default orc.compress.size, it has 168 columns, nearly all DECIMAL(38,x) datatype, and is partitioned on 3 columns. The source table for the insert-select operation is Avro with 165 columns, all string datatype. The dynamic partition insert would generate 3019 partitions if it were to succeed. I know this is a lot of partitions and wouldn't expect this to fit inside an 8G LLAP daemon, but the point of the question is to be able to as accurately as possible predict how much memory this insert-select operation will take with the information we have above. I've obtained a heapdump using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/hadoop/yarn/log/heapdump on the LLAP daemon and am running it through jhat. I will post the results when I have them. Thanks, Steve ... View more