CDH 6.3.2 Impala Issue(Error:N6apache6thrift9transport19TTransportExceptionE),Please Help me

[#1]

• I have a small CDH cluster for data processing. The cluster spans two computer rooms and the DataNodes are scattered in these two computer rooms. The tasks of MR and Spark are normal, but Impala always prompts exceptions, and the exception error is as follows:

• I0624 12:32:46.099220 28496 coordinator-backend-state.cc:472] Cancel query_id= 6f4bb40bcc1c566d:31c371cd00000000 could not do rpc to 1.0.0.56:27099 failure: Cancel() RPC failed: Timed out: CancelQueryFInstances RPC to 1.0.0.56:27099 timed out after 10.000s (SENT)
  I0624 12:32:46.099277 28496 coordinator-backend-state.cc:445] Sending CancelQueryFInstances rpc for query_id=6f4bb40bcc1c566d:31c371cd00000000 backend=1.0.0.57:27099
  I0624 12:32:46.109216 28496 coordinator-backend-state.cc:445] Sending CancelQueryFInstances rpc for query_id=6f4bb40bcc1c566d:31c371cd00000000 backend=1.0.0.58:27099
  I0624 12:32:46.115186 28496 coordinator-backend-state.cc:445] Sending CancelQueryFInstances rpc for query_id=6f4bb40bcc1c566d:31c371cd00000000 backend=1.0.0.59:27099
  I0624 12:32:56.117004 28496 kudu-util.h:108] Cancel() RPC failed: Timed out: CancelQueryFInstances RPC to 1.0.0.59:27099 timed out after 10.000s (SENT)
  /var/log/impalad/impalad.ERROR File:Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
  E0624 10:03:00.376569 28289 logging.cc:147] stderr will be logged to this file.
  E0624 11:42:32.020143 17651 fe-support.cc:515] a144f708a7019692:088a512500000000] RPC Error: Client for worker07.center:26000 hit an unexpected exception: THRIFT_ETIMEDOUT, type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala23TPrioritizeLoadResponseE, send: done
• May I ask what may be the cause of this? I checked the network in advance. Is the network interchangeable?Thanks

[#2]

Hi @wbaihp ,

1. Is this issue with all the queries or any few queries?

2. Please attach the query profile and its coordinator logs to have a look.

[#3]

Looking at the error, it looks like RPC call are timing out and there can be multiple reasons for the same, to get the RCA on this we would require your impala profile and the coordinator logs covering the issue time.

[#4]

@wbaihp, has any of the replies helped resolve your issue? If so, can you please mark the appropriate reply as the solution? It will make it easier for others to find the answer in the future.