Support Questions

Find answers, ask questions, and share your expertise

Who agreed with this topic

Impala Memory limit exceeded despite setting memory limit

avatar
Explorer

Despite executing a query with a memory limit set(say 5GB) and the impala deamon having sufficient memory, the query gets cancelled owing the error 'memory exceeded'.

 

High level error as below : 

 

queryMemory limit exceeded: Error occurred on backend ***** by fragment e4a777f7045b132:53b698be0000004c Memory left in process limit: 59.51 GB Memory left in query limit: -218.77 KB Query(e4a777f7045b132:53b698be00000000): memory limit exceeded. Limit=5.00 GB Reservation=3.93 GB ReservationLimit=4.00 GB OtherMemory=1.07 GB Total=5.00 GB Peak=5.00 GB Unclaimed reservations: Reservation=241.94 MB OtherMemory=0 Total=241.94 MB Peak=608.38 MB Fragment e4a777f7045b132:53b698be0000007f: Reservation=0 OtherMemory=220.02 KB Total=220.02 KB Peak=1.14 MB.

 

Note : the query was executed with mem_limit  of 5GB. I believe that even if it exceeds the query limits, it should rather spill data to disk than cancel the query. 

 

Can someone reason this error. Despite process memory good enough for execution, the query fails.

 

Who agreed with this topic