Impala - Couldn't open transport, Connection reset by peer

[#1]

 Hello -

We noticed an on-going issue at times, where Impala queries will be receive this type of message:

     Query Status: Couldn't open transport for <hostname>:22000 (SSL_connect: Connection reset by peer)

We are running CDH 5.7.3 and Impalad verison is 2.5.0.

When we see this, i can look at the webui for the impalad host and I usually see a query that in the "CREATED" state, but is not running...and typically these queries are from days before.  

I also notice that the Last Event will indicate something like "Ready to start 47 remote fragments".  I try to cancel (esp if the query is 2 or 3 days old) and i cannot cancel it and get this message: Error: Query not yet running

Seems the only way to clear the query is to reset the Impalad node.  That seems like bad way to resolve this issue.

Has anyone faced this issue before and have any thoughts/suggestions?

thanks...

[#2]

Were you able to find a resolution to this ?

[#3]

Hello @dwill @Srivatsan 

CDH 5.7.3 is a very old version and there has been lot of fixes post that.

Again coming to that error. Generally when you see the query in created state, we need to check where exactly the query is waiting. For example, query can be in created state if it is not able to fetch metadata from catalog server which is needed for submitting the query. It can also be in created state if the resources are less and query is queued.

[#4]

Thanks @tusharkathpal . Is there a easy way to determine the cause of the failure (catalog server vs resource crunch), given that it happens fairly inconsistent in my case.