Member since
09-07-2018
3
Posts
0
Kudos Received
0
Solutions
09-07-2018
05:47 AM
Hi there, We are having the same issue. We are trying to compute incremental stats for a table, even specifing the partitions. The data volume is 900Mb, in 4 files, around 2.000 columns and saved as parquet. And we got: Status: Memory limit exceeded: The memory limit is set too low to initialize spilling operator (id=1). The minimum required memory to spill this operator is 0. Error occurred on backend ----- by fragment 494c7492338f9bc1:33554c2400000001 Memory left in process limit: 139.01 GB Query(494c7492338f9bc1:33554c2400000000): Total=2.14 KB Peak=2.14 KB Fragment 494c7492338f9bc1:33554c2400000000: Total=8.00 KB Peak=8.00 KB EXCHANGE_NODE (id=4): Total=0 Peak=0 DataStreamRecvr: Total=0 Peak=0 PLAN_ROOT_SINK: Total=0 Peak=0 CodeGen: Total=0 Peak=0 Block Manager: Limit=111.32GB Total=0 Peak=0 Fragment 494c7492338f9bc1:33554c2400000004: Total=546.12 KB Peak=546.12 KB AGGREGATION_NODE (id=3): Total=546.12 KB Peak=546.12 KB Exprs: Total=508.00 KB Peak=508.00 KB EXCHANGE_NODE (id=2): Total=0 Peak=0 DataStreamRecvr: Total=0 Peak=0 Fragment 494c7492338f9bc1:33554c2400000001: Total=546.12 KB Peak=546.12 KB AGGREGATION_NODE (id=1): Total=546.12 KB Peak=546.12 KB Exprs: Total=508.00 KB Peak=508.00 KB HDFS_SCAN_NODE (id=0): Total=0 Peak=0 This propertie is set as follow: -inc_stats_size_limit_bytes=275090541379584000 Can not believe that with such small amount of data, Impala is not be able to make the maths. I do not know if it has something to see with this formula (but still, it has more than enough to go through): 400 bytes of metadata * column * partition Any help here will be great. Thanks in advance! 🙂
... View more