FunctionContextImpl::AllocateLocal's allocations exceeded memory limits

[#1]

Hello,

I see that the following error is triggered regularly on my cluster:

> Memory limit exceeded

> FunctionContextImpl::AllocateLocal's allocations exceeded memory limits

Such error, appears when :

• requests address a very large number of tuples
• requests seems to use only built-in UDF/UDAF like SUM, COUNT, or DISTINCT

I see that if I reran the request, the request then succeed almost everytime. However, i have haproxy in use with lot of impalads behind it.

Questions :

1) Is FunctionContextImpl::AllocateLocal called by theses built-in UDAF ? It is a typical error that means that impalad is running out of memory for the current request ? If yes, increasing memory of each impalad could solve the problem easily ?

2) Could the problem be related to memory leaks ?

I mean that I have also self-made C++ UDF that are in use with others Impala requests. Such requests (which address smaller number of tuples) succeed. However in such UDFs I use FunctionContextImpl::Allocate/Free ( not the AllocateLocal one ), and the StringVal constructor with context parameter.

So basically, if a memory leak is actually happening in the self-made UDF, could it be related to the previous error ? I mean the "AllocateLocal's allocations exceeded memory" error that occurs on request which are not using the self-made UDF.

Thanks !

[#2 · Accepted solution]

Hi @Plop564,

  Thanks for the quality question.

1) It's also used by a lot of Impala builtin functions (we implement many with the exact same interface as UDFs).

2) It could be a bug in a UDF/UDAF. It's also possible that there's a bug in Impala where Impala isn't cleaning up the local allocations during some operation. E.g. in older versions if you ordered by a UDF that allocated memory, then a lot of that memory was only cleaned up at the end of the sort.

It's possible that you could be allocating a lot of memory with FunctionContextImpl::Allocate() and Free() as well. The memory allocated in that way counts against the same limits as AllocateLocal() for the most part.

[#3]

Hi @Tim Armstrong

  Thanks for the quality answer 🙂

As you mention, we don't use in production the lastest Impala version, so it is indeed possible that there are bugs in Impala or UDF/UDAF. I will check the changelog and evaluate possibles issues on thoses very large requests.

Regarding our self-made UDF, the good thing is, after reviewing our logs history, such error was also triggered before the deployement of such UDF. So if there are some memory leaks currently, it might be then unrelated to our work, and it might be a minor issue as we have just to wait to upgrade the cluster (and Impala version)

Otherwise, many thanks for implementation details you give me, it helps to better understand !