Reply
Explorer
Posts: 7
Registered: ‎11-16-2016

Impala JDBC Connection string memory limit exceeded

Hi Team,

 

Have successfully created and tested JDBC connection string with MEM_LIMIT = 4gb on cognos.

 

Dynamic resource on the impala are set with max memory limit to 570 GB and default query limit set to 570 MB.

 

Have created 10 reports on cognos with this connection string, observed that in certain cases all reports are honoring 4 gb memory limit in the query plan. In certain cases, only certain reports are using the set 4gb memory limit but other reports are using default memory query limit i.e., 570 MB on the pool in the query.

 

Is there any logic or override logic in the impala code not to honor set memory limit 4GB and using default memory limit i.e. 570 MB on the JDBC connection string? Please advise.

 

Using CDH 5.7 version and only following 10 reports are running on this 570 GB resource pool.

 

Thanks,

Prashanth.

Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: Impala JDBC Connection string memory limit exceeded

 Where are you seeing that the memory limit isn't honoured?

 

You mention the plan - if you're looking at the memory estimates in the query plans do not factor in memory limits, they're an estimate of how much memory the query requires to execute without spilling to disk. 

 

The memory limit is enforced at runtime. If you look at "Per Node Peak Memory Usage" in the profile, that should not (significantly) exceed the query memory limit. You can see the memory limit (if there is one) in the query options:

 

Query Options (non default): MEM_LIMIT=4294967296

 

Explorer
Posts: 7
Registered: ‎11-16-2016

Re: Impala JDBC Connection string memory limit exceeded

Thanks for the response Tim.

 

In Impalad Daemon, under the query header, in the profile tab, have seen that query thread is not honoring the mem_limt 4G set on the JDBC connection string.

 

Here is the details on cognos,

 

Connection String:-

 

jdbc:impala://localhost:18000/default2;AuthMech=3;

UID=cloudera;PWD=cloudera;MEM_LIMIT=4294967296;REQUEST_POOL=myPool

 

Report on Day-1, Considering the 4GB mem_limt

 

In Query Profile,

 

Query (id=fgert53423436435:uiewryehfdbf345)

  Summary:

       :

       :

    Query Status:

   Completed

       :

       :

     Query Options (non default): MEM_LIMIT=4294967296

     Plan:

------------------

  Estimated Per-Host Requirements: Memory=45.63 GB Vcores=10

======================================================================= 

Same report on some other day, Considering the 570 MB mem_limt

 

Query (id=d4253643dfgsd:erteyu3476546)

  Summary:

       :

       :

    Query Status:

  Memory limit exceeded

       :

       :

     Query Options (non default): MEM_LIMIT=597688320

     Plan:

------------------

  Estimated Per-Host Requirements: Memory=45.63 GB Vcores=10

 

Please advise, does Impala code override the run time mem_limt set on the connection string, in case if there is no enough RAM exist on the Nodes with default Mem_limit i.e. 570 MB?

 

Thanks,

Prashanth.

 

 

Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: Impala JDBC Connection string memory limit exceeded

I think you missed this part of my earlier reply: 'The memory limit is enforced at runtime. If you look at "Per Node Peak Memory Usage" in the profile, that should not (significantly) exceed the query memory limit.'

Explorer
Posts: 7
Registered: ‎11-16-2016

Re: Impala JDBC Connection string memory limit exceeded

Yeah, have seen that information earlier. Yes, Per Node Peak Memory Usage is not exceeding the Mem Limit. But actually, my question was, why Set MEM_LIMIT to 4GB is getting override with 570 MB mem_limit in the query profile.

One of the fragment was expecting 2 GB but memory limit is getting set with 570 MB due to which my process are getting halted though the MEM_LIMIT at the client is set to 4GB.

Is this expected behaviour of the Impala, based on the available memory on the node during the runtime and it overrides the mem_limit set by the user at the client? Please advise.

 

Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: Impala JDBC Connection string memory limit exceeded

Oh ok, I see. No, Impala doesn't adjust memory limits like that. The memory limit is always determined by query options, which can either be set explicitly or come from the resource pool. Is it possible that you overrode the mem_limit query option sometime earlier in the session?

Explorer
Posts: 7
Registered: ‎11-16-2016

Re: Impala JDBC Connection string memory limit exceeded

Hi have set memory limit in the jdbc connection string itself to 4GB .....out of 10 reports only 8 reports are considering 4gb and remaining 2 reports are considering 570 mb i.e, one on the resource pool....please advise how to resolve this issue.....
Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: Impala JDBC Connection string memory limit exceeded

That is strange - is there any difference between the queries or connection strings between the two queries? The expected behaviour is that the query option in connection string should take precedence over the resource pool default.

 

 

What happens if you execute the "set mem_limit=4gb" statement before running the query (instead of using the connection string.

Explorer
Posts: 6
Registered: ‎10-04-2017

Re: Impala JDBC Connection string memory limit exceeded

I'm also having this issue. The SET MEM_LIMIT=x; works as a query, but if you put it in the jdbc connection string, it is not obeyed.

 

Unfortunately with sqoop -- the connection string seems be the only place you can put it.

 

Announcements