Reply
Expert Contributor
Posts: 113
Registered: ‎02-15-2016

impala does not work if any of the impala server is down

Hi ,

I am trying to understand this situation .

 

Lets say i have 5 impalad sever and if anyone one them is down impala queries will get timeout error or will not execute . what is the reason behind this ? 

 

Cloudera Employee
Posts: 355
Registered: ‎07-29-2015

Re: impala does not work if any of the impala server is down

If a daemon goes down suddently then you will see query failures. After a period Impala should detect that the other daemon is down and stop routing queries to it.

 

One way to debug is to look at http://impala-host:25000/backends and see what other servers it thinks are alive.

 

I'm not sure what timeout errors you're referring to. Is it possible that the daemon is up and there are just connectivity issues?

Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: impala does not work if any of the impala server is down

let me explain with example 

 

top - 10:57:50 up 58 days, 37 min, 7 users, load average: 257.91, 256.55, 253.75
Tasks: 1322 total, 1 running, 1302 sleeping, 0 stopped, 19 zombie
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 82.3 id, 17.5 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 26370188+total, 12234492 free, 51030360 used, 20043704+buff/cache
KiB Swap: 8388604 total, 5778116 free, 2610488 used. 21161840+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
39762 root 39 19 1411748 35056 3712 S 6.0 0.0 95:15.54 receptor
16806 impala 20 0 46.612g 5.321g 28840 S 1.0 2.1 732:22.32 impalad
22047 root 20 0 158908 3472 1540 R 1.0 0.0 0:00.08 top
12285 hbase 20 0 26.438g 0.018t 13132 S 0.7 7.5 1775:40 java
65 root 20 0 0 0 0 S 0.3 0.0 58:05.24 rcuos/14
11525 yarn 20 0 4991204 1.977g 14540 S 0.3 0.8 35206:02 java
12416 hbase 20 0 114580 2156 692 S 0.3 0.0 218:57.29 hbase.sh
17528 complia+ 20 0 3182376 843428 27068 S 0.3 0.3 8:00.97 java
17888 complia+ 20 0 3181380 842192 26796 S 0.3 0.3 8:35.77 java
20861 complia+ 20 0 3195300 882332 26860 S 0.3 0.3 8:51.63 java
32800 hive 20 0 3448532 1.047g 25244 S 0.3 0.4 6:42.25 java

 

this server hot hung with more load . since this one of the impalad server ,now imapad is not working on this server which cause whole impala service to hang . 

 

i am thinking why  issue in one impalad server causing whole impala service to hang . 

 

after restarting this hung server impala works fine .

Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: impala does not work if any of the impala server is down

guys any thought /suggestion on this
Cloudera Employee
Posts: 355
Registered: ‎07-29-2015

Re: impala does not work if any of the impala server is down

Hi @MSharma it does sound like a bug but it's hard to diagnose without more info.

 

Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: impala does not work if any of the impala server is down

Let me know what info you need , i can share .
Announcements