Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Kerberos authentication error with keytab

Explorer

Hello,

 

I've got a problem with the authentication of Kerberos using the Keytab, when I try to start any instance of HDFS service I keep getting the next error.

 

org.apache.hadoop.security.KerberosAuthException: Login failure for user: hdfs/<fqdn>@<REALM.COM> from keytab hdfs.keytab javax.security.auth.login.LoginException: Message stream modified (41)

I did not found any satisfactory answer for this problem, and the principals authenticates very well using that keytab file through kinit command.

 

Thank you in advance.

13 REPLIES 13

New Contributor

Hi,

 

check that the Domain name in your krb5.conf is in uppercase:

 

 

default_realm = EXAMPLE.COM

EXAMPLE.COM = {
kdc = domaincontroller.example.com
admin_server = domaincontroller.example.com
default_domain = EXAMPLE.COM
}

.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

 

regards

Explorer

Yes, the realm name is in uppercase, the same as in the examples

Champion

@rrodriguez

 

1. Get the node has keytab. Ex: I am using the node which belongs to impala daemon.
2. Go to /var/run/cloudera-scm-agent/process and ls, it will list the process
3. cd xxxxxx-impala-IMPALAD .. ## Run ls cmd and make sure it has impala.keytab
4. klist -kt impala.keytab ## This will list all the available and valid principals
5. kinit -kt impala.keytab <copy paste the valid principal from the above step>
6. klist ## make sure kinit initiated

Explorer

Sorry for the late response.

 

I did that and it worked with kinit and an imported keytab

 

Thank you

Champion

@rrodriguez happy to know that it worked!!

Explorer

Sorry, worked the kinit command, Cloudera keeps giving the first mentioned exception.

Champion

@rrodriguez

 

Is it? my bad, i didn't get it... Did you get a chance to follow 'all' the steps that i've mentioned? if so, were you able to run all the steps successfully?

Explorer

@saranvisa

 

Yes I've done all the steps in multiple ocasions, kinit command works fine with the keytabs imported but HDFS continues writing that error in logs.

Explorer

Hello @saranvisa

 

I tested it again after doing a regenerate keytabs and when doing the klist -kt I got the next message.

 

# klist -kt hdfs.keytab
Keytab name: FILE:hdfs.keytab
klist: Unsupported key table format version number while starting keytab scan

 

This is not the same for other keytab files in other directories into /var/run/cloudera-scm-agent/process just for some of them.

 

Any idea of what's happening? Why some processes are getting empty keytab files? I don't understand.

 

Thank you for the help

Explorer

Hi again @saranvisa,

 

I checked the logs and I saw that the error that I was getting on starting a service was caused from a certain process so I got in that directory and looked for the error on hdfs.keytab. When doing the klist -kt hdfs.keytab I got the principals list, tried to make a kinit with one of them and it worked well.

 

What I've seen is that the imported keytabs I was trying to klist were some old keytab files, modified few weeks ago, and the logs gave me the clue on which directory test the keytab files.

 

So we are at the same point, seems that krb5-workstation commands work fine, keytabs were generated right and the service keeps outputing the same error again and again.

 

Some more ideas to test?

 

Thank you

Explorer

We surpassed the error just configuring Cloudera to authenticate to a local KDC, we were using a KDC provided by WSO2, this problem got solved but not with the scenario it appeared first.

New Contributor

Do you wanted to check the proper Authentication is Happenning between Cluster and AD, make sure port,

Cloudera Employee

When you get below error message when doing kinit using a keytab file

klist: Unsupported key table format version number while starting keytab scan

Make sure that keytab file is not of zero byte 
e.g This is Zero byte keytab file and you will get the above error when trying to do kinit with it

-rw------- 1 cloudera-scm cloudera-scm 0 Aug 30 12:15 ./32-cloudera-mgmt-SERVICEMONITOR/cmon.keytab

A good keytab file will have non-zero size e.g. 778 for the below file 

-rw------- 1 cloudera-scm cloudera-scm 778 Oct 12 05:21 ./150-cloudera-mgmt-SERVICEMONITOR/cmon.keytab

 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.