Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and realm

[#1]

As per-title, the is a new Cloudera Manager 5.15 setting for Impala called "Impala Daemon KRPC Port" which is a setting for parm called "krpc_port" and is set to port 27000

We use a mix of SSSD, Kerberos, Sentry and Active Directory for our clusters. Jumping from CDH 5.13.0 to 5.15.0 in one of our clusters results in Impala queries failing with...

• TransmitData() to 10.0.1.204:27000 failed: Not authorized: Client connection negotiation failed: client connection to 10.0.1.204:27000: Cannot determine realm for numeric host address

Rolling back to CDH 5.13.0 confirms all ok again, rolling forward to CDH 5.15.0 breaks this. 

The query error appears in both Hue and Impala Shell. Of note you can still traverse though the use database commands and show tables OK, just when quering the data.

I've checked everything I can think of so far, /etc/hosts is fine, along with /etc/krb5.conf (and importantly haven't changed for years), reverse DNS digs work OK, Kerberos tickets are issued etc. 

Just throwing this out there in-case someone else is/has experienced this issue.

[#2]

To add some extra detail, started seeing some malformed packets with Thrift when attempting to CALL ExecQueryFInstances on what looks like port 22000. This seems to be between two Impala Daemons (see attached screen shot, the very first malformed packet is highlight, there are further ones including the final error message about numeric realm, but of not NOT on port 22000)

Doing a Google wordsearch the terms "ExecQueryFInstances" and "malformed" only has this...

http://mail-archives.apache.org/mod_mbox/impala-commits/201804.mbox/%3C009cd324f41647e388b31b7066cd6...

I am using the following parcels downloaded from the CDH5 site

5.15.0-1.cdh5.15.0.p0.21

[#3 · Accepted solution]

Hi @mcalnd,

We recently found a bug in 5.15, which we've tracked here:
https://issues.apache.org/jira/browse/IMPALA-7298

I wonder if your error is a different symptom of the same bug. In krb5.conf, are the following flags set to true?

"rdns=true" & "dns_canonicalize_hostname=true"

If they're false, then we hit a known bug as mentioned in the above JIRA.

- Sailesh

[#4]

@saileshmukil

Thanks for this, we had 'rdns = false' in the krb5.conf, setting to true resolved this issue.

Seems that this error (realm message) is another by-product of the bug. I've spent about 3 man days troubleshooting this issue so I'm glad I can move on!