impala error - ERROR: ExecPlanRequest rpc query_id

[#1]

I get the below error on impala 1.2.3 version. I did the recycle of all the impalad on cluster but still I get the below error when use “select * tablename” but same select statement works from hive.

ERROR: ExecPlanRequest rpc query_id=8641141086f3ab8e:8ff3926e0f221e91 instance_id=8641141086f3ab8e:8ff3926e0f221e95 failed: ECONNRESET

I have not seen this error before but I ran out of disk space on couple of nodes and I think it caused this issue.

Thanks,

Ram

[#2]

Hi Ram,
Are you able to successfully run "select 1" on every impalad node? If you
are running CM, can you check the Cluster Health to see if there are any
warnings or errors reported? It will be hard to debug your issue further
unless we have more of your log file.

Thanks,
Lenni

[#3]

Hi Lenni,

I get the same error when I tried to execute the "select 1 from table_name" from impala-shell, however same select statement works from hive on all the nodes.
Error:

ERROR: ExecPlanRequest rpc query_id=7c4cc62c0d4c147f:e23ddf1caf162aba instance_id=7c4cc62c0d4c147f:e23ddf1caf162abc failed: ECONNRESET

[#4]

What if you run "select 1" (leave out the FROM clause)?

Thanks,
Lenni

[#5]

Select 1 works on all the cluster nodes - pls see below
[CHAS2T3ENDC03.ops.tiaa-cref.org:21000] > select 1 ;
Query: select 1
+---+
| 1 |
+---+
| 1 |
+---+
Returned 1 row(s) in 0.14s
[CHAS2T3ENDC03.ops.tiaa-cref.org:21000] > select 1 from vpnagg;
Query: select 1 from vpnagg
ERROR: ExecPlanRequest rpc query_id=2e432968a16d64b0:b682c60a0df3a799 instance_id=2e432968a16d64b0:b682c60a0df3a79b failed: ECONNRESET
[CHAS2T3ENDC03.ops.tiaa-cref.org:21000] >

[#6 · Accepted solution]

Hello All,

It was because of port conflict with other services running the server (impala daemon backend port be_port) 22000 has conflict on new services installed on linux server.

The strange think is CM shows all "GREEN"  and all the process has been started but when you look at the log file it will show the following error -

+ exec /usr/lib/impala/../../bin/impalad --flagfile=/var/run/cloudera-scm-agent/process/2050-impala-IMPALAD/impala-conf/impalad_flags
E0508 18:20:58.253557 16729 thrift-server.cc:157] ThriftServer 'backend' (on port: 22000) exited due to TException: Could not bind: Transport endpoint is not connected
E0508 18:20:58.254359 16152 thrift-server.cc:146] ThriftServer 'backend' (on port: 22000) did not start correctly

please make life easier 🙂

Thanksm

Ram

[#7]

Hi,
I have filed https://issues.cloudera.org/browse/IMPALA-990 to track this
issue. Impala should fail to start if any of the thrift servers cannot
start.

Thanks,
Lenni

[#8]

Hi,

We are facing the same exception on a running , on one of the nodes. We are running on Impala version 2.3. As per above update this issue should have resolved in Impala 1.4. Please help as it is impacting stability of cluster. All nodes stops responding even though problem is only with one node.

Exception:

E0602 09:30:30.544390 18138 logging.cc:120] stderr will be logged to this file.

E0602 09:30:39.605379 18220 thrift-server.cc:160] ThriftServer 'backend' (on port: 22000) exited duelp e to TException: Could not $

E0602 09:30:39.605533 18138 thrift-server.cc:149] ThriftServer 'backend' (on port: 22000) did not start correctly

E0602 09:30:39.605597 18138 impalad-main.cc:76] ThriftServer 'backend' (on port: 22000)

[#9]

Hi scotty603,



does the impalad on the affected node start up and keep running, despite
the ThriftServer exception? Have you checked that port 22000 is actually
unused prior to starting impala?



Cheers, Lars

[#10]

1- When we started impala service it started fine. Please see statestored.info file. I have removed my actual server name. And it was working fine. Suddenly on 02-Jun 07:53:09 it was unable to send heartbeat. And at 9:30 we see error I posted earlier about ThriftServer.

2- so nothing was running on port 22000.

namenode->statestored.info
I0506 14:46:35.929142 4306 statestore.cc:370] Registering: impalad@<node name which got impacted>:22000
I0506 14:46:35.929231 4306 statestore.cc:393] Subscriber 'impalad@<node name which got impacted>:22000' registered (registration id: fc490552ddd59468:113b9a75e4b55787)
I0602 07:53:09.326907 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:13.325565 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:17.321043 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:21.320971 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:25.320952 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:29.320972 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:33.320968 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:37.320973 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:41.316970 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:45.316970 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:49.316972 4153 statestore.cc:690] Unable to send heartbeat message to subscriber impalad@<node name which got impacted>:22000, received error: RPC timed out
I0602 07:53:49.316988 4153 statestore.cc:702] Subscriber 'impalad@<node name which got impacted>:22000' has failed, disconnected or re-registered (last known registration ID: impal$
I0602 10:09:16.522661 17361 statestore.cc:370] Registering: impalad@<node name which got impacted>:22000
I0602 10:09:16.522711 17361 statestore.cc:393] Subscriber 'impalad@<node name which got impacted>:22000' registered (registration id: f45a9be0d38a32c:63a32de772688380)