Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and realm

avatar
Explorer

As per-title, the is a new Cloudera Manager 5.15 setting for Impala called "Impala Daemon KRPC Port" which is a setting for parm called "krpc_port" and is set to port 27000

 

We use a mix of SSSD, Kerberos, Sentry and Active Directory for our clusters. Jumping from CDH 5.13.0 to 5.15.0 in one of our clusters results in Impala queries failing with...

 

  • TransmitData() to 10.0.1.204:27000 failed: Not authorized: Client connection negotiation failed: client connection to 10.0.1.204:27000: Cannot determine realm for numeric host address

Rolling back to CDH 5.13.0 confirms all ok again, rolling forward to CDH 5.15.0 breaks this. 

 

The query error appears in both Hue and Impala Shell. Of note you can still traverse though the use database commands and show tables OK, just when quering the data.

  

I've checked everything I can think of so far, /etc/hosts is fine, along with /etc/krb5.conf (and importantly haven't changed for years), reverse DNS digs work OK, Kerberos tickets are issued etc. 

 

Just throwing this out there in-case someone else is/has experienced this issue.

 

1 ACCEPTED SOLUTION

avatar
Contributor

Hi @mcalnd,

 

We recently found a bug in 5.15, which we've tracked here:
https://issues.apache.org/jira/browse/IMPALA-7298

I wonder if your error is a different symptom of the same bug. In krb5.conf, are the following flags set to true?

"rdns=true" & "dns_canonicalize_hostname=true"

 

If they're false, then we hit a known bug as mentioned in the above JIRA.

 

- Sailesh

View solution in original post

3 REPLIES 3

avatar
Explorer

To add some extra detail, started seeing some malformed packets with Thrift when attempting to CALL ExecQueryFInstances on what looks like port 22000. This seems to be between two Impala Daemons (see attached screen shot, the very first malformed packet is highlight, there are further ones including the final error message about numeric realm, but of not NOT on port 22000)

 

Doing a Google wordsearch the terms "ExecQueryFInstances" and "malformed" only has this...

http://mail-archives.apache.org/mod_mbox/impala-commits/201804.mbox/%3C009cd324f41647e388b31b7066cd6...

 

I am using the following parcels downloaded from the CDH5 site

5.15.0-1.cdh5.15.0.p0.21


PublicForumCapture.JPG

avatar
Contributor

Hi @mcalnd,

 

We recently found a bug in 5.15, which we've tracked here:
https://issues.apache.org/jira/browse/IMPALA-7298

I wonder if your error is a different symptom of the same bug. In krb5.conf, are the following flags set to true?

"rdns=true" & "dns_canonicalize_hostname=true"

 

If they're false, then we hit a known bug as mentioned in the above JIRA.

 

- Sailesh

avatar
Explorer

@saileshmukil

 

Thanks for this, we had 'rdns = false' in the krb5.conf, setting to true resolved this issue.

 

Seems that this error (realm message) is another by-product of the bug. I've spent about 3 man days troubleshooting this issue so I'm glad I can move on!