Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and realm

Solved Go to solution

Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and realm

Explorer

As per-title, the is a new Cloudera Manager 5.15 setting for Impala called "Impala Daemon KRPC Port" which is a setting for parm called "krpc_port" and is set to port 27000

 

We use a mix of SSSD, Kerberos, Sentry and Active Directory for our clusters. Jumping from CDH 5.13.0 to 5.15.0 in one of our clusters results in Impala queries failing with...

 

  • TransmitData() to 10.0.1.204:27000 failed: Not authorized: Client connection negotiation failed: client connection to 10.0.1.204:27000: Cannot determine realm for numeric host address

Rolling back to CDH 5.13.0 confirms all ok again, rolling forward to CDH 5.15.0 breaks this. 

 

The query error appears in both Hue and Impala Shell. Of note you can still traverse though the use database commands and show tables OK, just when quering the data.

  

I've checked everything I can think of so far, /etc/hosts is fine, along with /etc/krb5.conf (and importantly haven't changed for years), reverse DNS digs work OK, Kerberos tickets are issued etc. 

 

Just throwing this out there in-case someone else is/has experienced this issue.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and rea

Cloudera Employee

Hi @mcalnd,

 

We recently found a bug in 5.15, which we've tracked here:
https://issues.apache.org/jira/browse/IMPALA-7298

I wonder if your error is a different symptom of the same bug. In krb5.conf, are the following flags set to true?

"rdns=true" & "dns_canonicalize_hostname=true"

 

If they're false, then we hit a known bug as mentioned in the above JIRA.

 

- Sailesh

3 REPLIES 3

Re: Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and rea

Explorer

To add some extra detail, started seeing some malformed packets with Thrift when attempting to CALL ExecQueryFInstances on what looks like port 22000. This seems to be between two Impala Daemons (see attached screen shot, the very first malformed packet is highlight, there are further ones including the final error message about numeric realm, but of not NOT on port 22000)

 

Doing a Google wordsearch the terms "ExecQueryFInstances" and "malformed" only has this...

http://mail-archives.apache.org/mod_mbox/impala-commits/201804.mbox/%3C009cd324f41647e388b31b7066cd6...

 

I am using the following parcels downloaded from the CDH5 site

5.15.0-1.cdh5.15.0.p0.21


PublicForumCapture.JPG

Re: Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and rea

Cloudera Employee

Hi @mcalnd,

 

We recently found a bug in 5.15, which we've tracked here:
https://issues.apache.org/jira/browse/IMPALA-7298

I wonder if your error is a different symptom of the same bug. In krb5.conf, are the following flags set to true?

"rdns=true" & "dns_canonicalize_hostname=true"

 

If they're false, then we hit a known bug as mentioned in the above JIRA.

 

- Sailesh

Re: Impala Daemons in CDH 5.15 introduces KRPC port 27000 - Issues with queries TransmitData and rea

Explorer

@saileshmukil

 

Thanks for this, we had 'rdns = false' in the krb5.conf, setting to true resolved this issue.

 

Seems that this error (realm message) is another by-product of the bug. I've spent about 3 man days troubleshooting this issue so I'm glad I can move on!