Kerberos authentication error with keytab

[#1]

Hello,

I've got a problem with the authentication of Kerberos using the Keytab, when I try to start any instance of HDFS service I keep getting the next error.

org.apache.hadoop.security.KerberosAuthException: Login failure for user: hdfs/<fqdn>@<REALM.COM> from keytab hdfs.keytab javax.security.auth.login.LoginException: Message stream modified (41)

I did not found any satisfactory answer for this problem, and the principals authenticates very well using that keytab file through kinit command.

Thank you in advance.

[#2]

Hi,

check that the Domain name in your krb5.conf is in uppercase:

default_realm = EXAMPLE.COM

EXAMPLE.COM = {
kdc = domaincontroller.example.com
admin_server = domaincontroller.example.com
default_domain = EXAMPLE.COM
}

.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

regards

[#3]

Yes, the realm name is in uppercase, the same as in the examples

[#4]

@rrodriguez

1. Get the node has keytab. Ex: I am using the node which belongs to impala daemon.
2. Go to /var/run/cloudera-scm-agent/process and ls, it will list the process
3. cd xxxxxx-impala-IMPALAD .. ## Run ls cmd and make sure it has impala.keytab
4. klist -kt impala.keytab ## This will list all the available and valid principals
5. kinit -kt impala.keytab <copy paste the valid principal from the above step>
6. klist ## make sure kinit initiated

[#5]

Sorry for the late response.

I did that and it worked with kinit and an imported keytab

Thank you

[#6]

@rrodriguez happy to know that it worked!!

[#7]

Sorry, worked the kinit command, Cloudera keeps giving the first mentioned exception.

[#8]

@rrodriguez

Is it? my bad, i didn't get it... Did you get a chance to follow 'all' the steps that i've mentioned? if so, were you able to run all the steps successfully?

[#9]

@saranvisa

Yes I've done all the steps in multiple ocasions, kinit command works fine with the keytabs imported but HDFS continues writing that error in logs.

[#10]

Hello @saranvisa

I tested it again after doing a regenerate keytabs and when doing the klist -kt I got the next message.

# klist -kt hdfs.keytab
Keytab name: FILE:hdfs.keytab
klist: Unsupported key table format version number while starting keytab scan

This is not the same for other keytab files in other directories into /var/run/cloudera-scm-agent/process just for some of them.

Any idea of what's happening? Why some processes are getting empty keytab files? I don't understand.

Thank you for the help

[#11]

Hi again @saranvisa,

I checked the logs and I saw that the error that I was getting on starting a service was caused from a certain process so I got in that directory and looked for the error on hdfs.keytab. When doing the klist -kt hdfs.keytab I got the principals list, tried to make a kinit with one of them and it worked well.

What I've seen is that the imported keytabs I was trying to klist were some old keytab files, modified few weeks ago, and the logs gave me the clue on which directory test the keytab files.

So we are at the same point, seems that krb5-workstation commands work fine, keytabs were generated right and the service keeps outputing the same error again and again.

Some more ideas to test?

Thank you

[#12]

We surpassed the error just configuring Cloudera to authenticate to a local KDC, we were using a KDC provided by WSO2, this problem got solved but not with the scenario it appeared first.

[#13]

Do you wanted to check the proper Authentication is Happenning between Cluster and AD, make sure port,

[#14]

When you get below error message when doing kinit using a keytab file

klist: Unsupported key table format version number while starting keytab scan

Make sure that keytab file is not of zero byte 
e.g This is Zero byte keytab file and you will get the above error when trying to do kinit with it

-rw------- 1 cloudera-scm cloudera-scm 0 Aug 30 12:15 ./32-cloudera-mgmt-SERVICEMONITOR/cmon.keytab

A good keytab file will have non-zero size e.g. 778 for the below file 

-rw------- 1 cloudera-scm cloudera-scm 778 Oct 12 05:21 ./150-cloudera-mgmt-SERVICEMONITOR/cmon.keytab