07-17-2018 06:49 AM - last edited on 07-17-2018 01:58 PM by cjervis
As per-title, the is a new Cloudera Manager 5.15 setting for Impala called "Impala Daemon KRPC Port" which is a setting for parm called "krpc_port" and is set to port 27000
We use a mix of SSSD, Kerberos, Sentry and Active Directory for our clusters. Jumping from CDH 5.13.0 to 5.15.0 in one of our clusters results in Impala queries failing with...
Rolling back to CDH 5.13.0 confirms all ok again, rolling forward to CDH 5.15.0 breaks this.
The query error appears in both Hue and Impala Shell. Of note you can still traverse though the use database commands and show tables OK, just when quering the data.
I've checked everything I can think of so far, /etc/hosts is fine, along with /etc/krb5.conf (and importantly haven't changed for years), reverse DNS digs work OK, Kerberos tickets are issued etc.
Just throwing this out there in-case someone else is/has experienced this issue.
07-17-2018 07:38 AM - edited 07-17-2018 08:19 AM
To add some extra detail, started seeing some malformed packets with Thrift when attempting to CALL ExecQueryFInstances on what looks like port 22000. This seems to be between two Impala Daemons (see attached screen shot, the very first malformed packet is highlight, there are further ones including the final error message about numeric realm, but of not NOT on port 22000)
Doing a Google wordsearch the terms "ExecQueryFInstances" and "malformed" only has this...
I am using the following parcels downloaded from the CDH5 site
07-17-2018 09:18 AM
We recently found a bug in 5.15, which we've tracked here:
I wonder if your error is a different symptom of the same bug. In krb5.conf, are the following flags set to true?
"rdns=true" & "dns_canonicalize_hostname=true"
If they're false, then we hit a known bug as mentioned in the above JIRA.
07-18-2018 12:24 AM
Thanks for this, we had 'rdns = false' in the krb5.conf, setting to true resolved this issue.
Seems that this error (realm message) is another by-product of the bug. I've spent about 3 man days troubleshooting this issue so I'm glad I can move on!