You're absolutely right - we use 10% as the default estimate for selectivity for scan predicates when we don't have a better estimate. One case where we have a better estimate is when the predicate is something like id = 100. In that case we can estimate that the selectivity is 1 / (num distinct values).   There's also some logic to handle combining the estimates when there are multiple conditions.   If you're curious, the code is here: https://github.com/apache/incubator-impala/blob/4db330e69a2dbb4a23f46e34b484da0d6b9ef29b/fe/src/main/java/org/apache/impala/planner/PlanNode.java#L518 ... View more

We added support for --ldap_password_cmd in Impala 2.5, which I think addresses this problem. See https://issues.cloudera.org/browse/IMPALA-1934   https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_shell_options.html ... View more

We could definitely improve some of the diagnostics there. My guess is that one node is either overloaded or has some kind of hardware issue - might be worth looking at the health and CPU/memory usage of different nodes to see if one stands out. ... View more

No I don't think you're missing any obvious optimisation. Yes we only use a single core per aggregation per Impala daemon. This is obviously not ideal so we have a big push right now to do full parallelization of every operator. ... View more

This is a known issue that we're actively working on: https://issues.cloudera.org/browse/IMPALA-2567   Your analysis is accurate. Part of the problem is the number of connections and the other part is the # of threads per connection. You may be able to change some operating system config settings to increase limits here (depending on which limit you're hitting).   In order to reduce the # of tcp conncetions required you would either need to reduce the number of fragments or reduce the number of node executing the query.   You could reduce the # of fragments by breaking up the query into smaller queries. E.g. creating temporary tables with the results of some of the subqueries.  You could also try executing the query on a single node by setting num_nodes=1 if the data size is small enough that this makes sense. I suspect your query is too large for that to work, but it's hard to tell (that's a huge query plan!) ... View more

I'm not the most knowledgeable person about this part of the code, but what you're saying is correct. One of the likely causes of long wait times is if the receiver is consuming data slower than the sender is sending it. ... View more

I think we already have an open issue for this that is being actively worked on https://issues.cloudera.org/browse/IMPALA-3210   I.e. we don't support it yet but it's in the pipeline. ... View more

The only way to do this with zero work would be to use a view. http://www.cloudera.com/documentation/enterprise/latest/topics/impala_create_view.html    Otherwise you do have to run the queries as part of your data pipeline as you mentioned.     ... View more

The main difference seems to be execution skew. In the second profile the max time for the join is over 3 minutes, compared to much lower in first profile. The average time isn't very different between the profiles. Probably the partitioning resulted in the data being distributed differently between the nodes, and for some reason that one node is slower. It doesn't look like it's necessarily processing more data, but maybe the node is more heavily loaded, or the data is somehow different.   Is the join condition something complicated? It's only processing a few thousand rows per second through the join, which is very low. ... View more

Ok, that's interesting.  The nonsense-looking symbol up the top is probably jitted code from your query, probably an expression or something like that.    36.50%   perf-18476.map                 [.] 0x00007f3c1d634b82                                                                                                                                              The other symbols like  GetDoubleVal() may be  what is calling this expensive function. It looks like it's possible ProbeTime in the profile that's the culprit.   Can you share the SQL for your query at all? I'm guessing that there's some expression in your query that's expensive to evaluate. E.g. joining on some complex expression, or doing some kind of expensive computation.   ... View more

Yes, that's right. It's enabled for some cases by default (broadcast joins) in Impala 2.5. To enable it for a wider category of joins you can set the query option runtime_filter_mode=global.  This setting will become the default in Impala 2.7 because of the performance benefits. ... View more

Impala limits the number of threads executing the query plan by design. Impala dynamically increases the number of scanner threads provided there are CPU and memory resources available - in this case it seems like there weren't CPU resource available.   If the machine is already busy adding more threads can actually decrease query throughput. ... View more