Member since
07-29-2015
535
Posts
140
Kudos Received
103
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4617 | 12-18-2020 01:46 PM | |
2895 | 12-16-2020 12:11 PM | |
1955 | 12-07-2020 01:47 PM | |
1517 | 12-07-2020 09:21 AM | |
989 | 10-14-2020 11:15 AM |
07-16-2018
01:37 PM
@AntonyNthanks for following up - glad to hear it!
... View more
06-29-2018
02:34 PM
It's on our roadmap - we're definitely aware of the gap and want to close it.
... View more
06-21-2018
11:00 AM
> Also, if I am using mem_lim query option while running query, will it bypass the "max mem" set at admission control setting? For example, I have set 400GB in max_mem and using mem_lim as 450 while running the query. The memory limit does not override the pool "Max Memory" - admission control won't admit queries if the total of their memory limit across all hosts exceeds the pool max memory. mem_limit is a per-host number while "Max Memory" is a cluster-wide number. > Another issue, if mem_lim is set at pool level, the number of queries that can be executed will be reduced right? Since mem_lim amount of RAM will be reserved for each query. Yeah, there's a trade-off between admitting more queries and reliably giving each query enough memory to run fast. One thing to keep in mind is that running more queries concurrently doesn't mean higher through - if you are running enough queries to max out CPU or disk, then admitting more concurrently won't improve throughput. > However, if set, Impala requests this amount of memory from each node, and the query does not proceed until that much memory is available. This can cause query failures since memory required for queries will vary from query to query. This depends a bit on the memory limit and version of Impala that you're running. If a query gets close to it's memory limit, two things can happen - it can slow down because of spilling or reduced # threads, or it can fail. If you have mem_limits set to reasonable values (e.g. 2GB+), that makes query failures much less likely because spilling will be reliable. In more recent versions of Impala, we've been reducing the changes of query failures in these cases. E.g. CDH5.13 had a lot of improvements for HASH JOIN, AGGREGATE and SORT.
... View more
06-20-2018
11:11 AM
Yes, exactly. If you want enforcement of memory consumption then that field needs to have a non-zero value. We're aware this could be easier and more intuitive. We're currently working on some improvements in this area.
... View more
06-19-2018
11:11 AM
@rlopezwe sometimes decide to include features in the CDH release that aren't in the base Apache Impala release or vice-versa. In this case the feature wasn't quite ready so we didn't include it. It looks like we need to fix the docs.
... View more
06-15-2018
11:08 AM
Did you set a default query memory limit for the pool? If you didn't, then there's no enforcement of memory consumption.
... View more
06-13-2018
07:03 PM
I expect it will be included in the 5.14.4 maintenance release. I'm not aware of a workaround aside from avoiding running on affected hardware without popcnt support.
... View more
06-12-2018
05:12 PM
@mauriciothat's great news! Thanks for the update. We do need to get this documented though.
... View more
06-06-2018
02:24 PM
The threshold is actually based on the per-host number of rows, so it's 2.6M / 30 = 86K in the example you provided
... View more