We have been using Impala for close to 1 year. Recently I have noticed that when running queries that for some queries that query finsishes quickly but the fetch never completes even for a very small acount of data.
Where should I check to resolve this outstanding issue?
I'd start by looking at the Query Timeline section of the runtime profile (It's near the top of the runtime profile).
The two counters that are interseting here are:
ClientFetchWaitTimer: this is the time waiting in the client.
RowMaterializationTimer: this is the time to construct the result rows
Even I am facing the same issue and already had a look at the query profile, it says Client fetch wait timer is high but I don't understand why is it High? How can I further narrowed down the root cause? What can I see the solution.
I am not sure if we have the solution for this?
ClientFetchWaitTimer measures the time when the coordinator had a rowbatch ready, but the client has not downloaded it. Having a high value here means that the client itself, or the network bandwidth to the client was the bottleneck of the whole process.
If you use impala-shell as client you can try the -B switch, which turns off pretty printing of the output and can significantly improve the throughput.
If you have a client app that you wrote then try to separate the fetching of the rows and processing of them and allocate more threads for processing.
In some cases RowsAvailable can be reported a little too early - there are a few specific cases where Impala *thinks* that it has a row ready to return, but actually was a bit too optimistic and it takes longer than expected to produce that first row. We have an improvement JIRA open to make this more intuitive https://issues.apache.org/jira/browse/IMPALA-7551