Skip to main content
bridgor
Gold ⭐️
April 26, 2019
Solved

Impala daemons constantly connecting/disconnecting

Hi everyone, 

 

Hoping someone can help. 

 

I'm getting this error when impalad connects: 

 

RPC error: Client for <hostname>:23000 hits an unexpected exception authorize: cannot authorize peer, type: N6apache6thrift9transport13TSSLException

 

I get a similar error when SSL is disabled. 

 

the connect/disconnect cycle repeats about every 1/2 second forever.

 

The log does include a message that the cert was successfully loaded so I don't think it's a problem with that, but I'm open to any suggestions.

 

Is the type: noted above literally referring to a connection to an HBase thrift server? 

 

Anyone know more about this specific error message? 

 

Thanks!

 

 

bridgor
bridgorAnswer
Gold ⭐️
May 6, 2019

Thanks to everyone who replied. It turns out that references to truststores and server keys, etc., and associated passwords may be cached, so when we changed these after moving the cluster, creating new cerrts and replacing the passwords in CDH was insufficient. 

 

So, after DELETING all fields containing passwords, cert locations, key locations, etc.,unchecking SSL, restarting the cluster, and adding the references back in, everything works. Uugghhh - who knew! 🙂

 

B

9 replies

AcharkiMed
Gold ⭐️⭐️⭐️
April 28, 2019

Hi @bridgor 

 

It looks like you have a firewall issue between your nodes.. try to check iptables.

Please share with us the impalad log files.

 

Good luck.

Translate
bridgor
bridgorAuthor
Gold ⭐️
April 28, 2019
Copy thanks. I shut down firewalld to test with the same result. Also, these servers are on the same switch.

I'll scrub logs and send what I am able to.

Thanks!
Translate