We have been having an issue with "Impala" in the environment where sometimes the connections between impala daemons gets refused and throws an error causing the queries to fail. This issue has been happening intermittently in the cluster at random times during the day. Especially Mondays and Wednesdays when there is huge load.
RPC client failed to connect: Couldn't open transport for p1i-hdp-srv09.example.com:22000 (connect() failed: Connection refused)
When I research on the error, I knew that the error " Connection refused" means there was no process listening on port 22000 on the peer
host p1i-hdp-srv09.example.com. The impalad on p1i-hdp-srv09.example.com either crashed or stopped at the moment.
I knew from this forum that I have to enable KRPC and got the below command online. But I am not sure what does it mean by Impala startup flags. Any clues?
Set -- use_krpc to true in the Impala startup flags.
The only way to get KRPC is to upgrade to CDH5.15 and greater, where it is enabled by default.
If you're using CM, some command line flags are either controlled by CM options. Others can be set via the command line safety valve options. But see above for what to do about KRPC.