We installed Cloudera Manager Standard 4.8.0 (with Impala 1.2.3-1.p0.97 and CDH 4.5.0) and we integrated it with Samba (SSSD). Using the krb5 client we have user authentication.
Most of the components work properly, but we have problems with Impala. Impala starts and seems to work properly, but now and then on some node the impala-shell hangs. We can execute one command, we get back the result, but afterwards we do not get back the command prompt. The only way to get out is to kill the process.
This behavior does not happen on one specific node, it happens on whichever node we use more often. If we restart the impala daemon on that node everything works properly.
We cannot find anything useful in the logs. We also tried to use strace to see what is going on, but no luck. The only indication is that the load is high on the given machine, but no other processes are taking up too much. When we stop the impala process the load immediately goes down.
We tried to reproduce the problem, but couldn’t find a good way to do this. It happens approximately once a week and it happens on random nodes (well the nodes we connect to with impala shell), so we cannot say that only one node is incorrectly configured.
We also have Hue installed and when we use impala from Hue, sometimes the whole web interface hangs for quite a while (i.e.. not only Impala, but Hive, etc. too). If we then connect to the node to which Hue is connecting for impala, we see that the impala-shell is hanging. If we wait long enough, Hue starts working again. ( so I assume it waits until for the timeout and then it starts again).
The certificates are renewable (I’ve triple checked). I am running out of idea of what I can test, especially that I cannot properly reproduce the problem. Any ideas what could be wrong or how I could debug this further?
Also, do you have any tips of how I could make impala log everything (i.e.. Warnings, Info, Errors) to one file? Having 3 files doesn’t give me a good overall view.
Thanks and regards,
Thanks for the reply, but our problem isn't really Hue. Hue only hangs when Impala hangs, so I would like to find a solution for the Impala problem and that will solve the Hue problem too. Anyway I did change the Hue server to CherryPy.
vis788, to assist in finding the root cause, would you mind giving us the following information:
- The version of Impala/CDH you're using.
- The impala-shell invocation you use
- A paste of your terminal when you see the 'hang'