Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala shell hangs

Impala shell hangs

New Contributor

We installed Cloudera Manager Standard 4.8.0 (with Impala 1.2.3-1.p0.97 and CDH 4.5.0) and we integrated it with Samba (SSSD). Using the krb5 client we have user authentication.

Most of the components work properly, but we have problems with Impala. Impala starts and seems to work properly, but now and then on  some node the impala-shell hangs. We can execute one command, we get back the result, but afterwards we do not get back the command prompt. The only way to get out is to kill the process. 

This behavior does not happen on one specific node, it happens on whichever node we use more often. If we restart the impala daemon on that node everything works properly. 

 

We cannot find anything useful in the logs. We also tried to use strace to see what is going on, but no luck. The only indication is that the load is high on the given machine, but no other processes are taking up too much. When we stop the impala process the load immediately goes down. 

 

We tried to reproduce the problem, but couldn’t find a good way to do this. It happens approximately once a week and it happens on random nodes (well the nodes we connect to with impala shell), so we cannot say that only one node is incorrectly configured.

 

We also have Hue installed and when we use impala from Hue, sometimes the whole web interface hangs for quite a while (i.e.. not only Impala, but Hive, etc. too). If we then connect to the node to which Hue is connecting for impala, we see that the impala-shell is hanging. If we wait long enough, Hue starts working again. ( so I assume it waits until for the timeout and then it starts again).

 

The certificates are renewable (I’ve triple checked). I am running out of idea of what I can test, especially that I cannot properly reproduce the problem. Any ideas what could be wrong or how I could debug this further?

 

Also, do you have any tips of how I could make impala log everything (i.e.. Warnings, Info, Errors) to one file? Having 3 files doesn’t give me a good overall view.  

 

Thanks and regards,

Tunde

4 REPLIES 4

Re: Impala shell hangs

New Contributor
When Hue hangs for everybody, this is very probably due to
http://gethue.tumblr.com/post/75493515493/solving-the-hue-2-x-hanging-problem

Re: Impala shell hangs

New Contributor

Thanks for the reply, but our problem isn't really Hue. Hue only hangs when Impala hangs, so I would like to find a solution for the Impala problem and that will solve the Hue problem too. Anyway I did change the Hue server to CherryPy.

Highlighted

Re: Impala shell hangs

New Contributor

my impala-shell hang while i am connecting it from kerberos. did you get any solution.

Re: Impala shell hangs

Cloudera Employee

vis788,  to assist in finding the root cause, would you mind giving us the following information:

 

- The version of Impala/CDH you're using.

- The impala-shell invocation you use

- A paste of your terminal when you see the 'hang'

Don't have an account?
Coming from Hortonworks? Activate your account here