question Re: Kerberos authentication error with keytab in Support Questions

Kerberos authentication error with keytab

rrodriguez — Wed, 15 Nov 2017 08:02:50 GMT

Hello,

I've got a problem with the authentication of Kerberos using the Keytab, when I try to start any instance of HDFS service I keep getting the next error.

org.apache.hadoop.security.KerberosAuthException: Login failure for user: hdfs/<fqdn>@<REALM.COM> from keytab hdfs.keytab javax.security.auth.login.LoginException: Message stream modified (41)

I did not found any satisfactory answer for this problem, and the principals authenticates very well using that keytab file through kinit command.

Thank you in advance.

Re: Kerberos authentication error with keytab

ancistrus17 — Wed, 15 Nov 2017 13:40:22 GMT

Hi,

check that the Domain name in your krb5.conf is in uppercase:

default_realm = EXAMPLE.COM

EXAMPLE.COM = {
kdc = domaincontroller.example.com
admin_server = domaincontroller.example.com
default_domain = EXAMPLE.COM
}

.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

regards

Re: Kerberos authentication error with keytab

rrodriguez — Wed, 15 Nov 2017 13:42:48 GMT

Yes, the realm name is in uppercase, the same as in the examples

Re: Kerberos authentication error with keytab

saranvisa — Wed, 15 Nov 2017 16:16:52 GMT

@rrodriguez

1. Get the node has keytab. Ex: I am using the node which belongs to impala daemon.
2. Go to /var/run/cloudera-scm-agent/process and ls, it will list the process
3. cd xxxxxx-impala-IMPALAD .. ## Run ls cmd and make sure it has impala.keytab
4. klist -kt impala.keytab ## This will list all the available and valid principals
5. kinit -kt impala.keytab <copy paste the valid principal from the above step>
6. klist ## make sure kinit initiated

Re: Kerberos authentication error with keytab

rrodriguez — Tue, 28 Nov 2017 16:17:04 GMT

Sorry for the late response.

I did that and it worked with kinit and an imported keytab

Thank you

Re: Kerberos authentication error with keytab

saranvisa — Tue, 28 Nov 2017 19:12:16 GMT

@rrodriguez happy to know that it worked!!

Re: Kerberos authentication error with keytab

rrodriguez — Wed, 29 Nov 2017 07:15:05 GMT

Sorry, worked the kinit command, Cloudera keeps giving the first mentioned exception.

Re: Kerberos authentication error with keytab

saranvisa — Wed, 29 Nov 2017 15:12:31 GMT

@rrodriguez

Is it? my bad, i didn't get it... Did you get a chance to follow 'all' the steps that i've mentioned? if so, were you able to run all the steps successfully?

Re: Kerberos authentication error with keytab

rrodriguez — Wed, 29 Nov 2017 16:01:11 GMT

@saranvisa

Yes I've done all the steps in multiple ocasions, kinit command works fine with the keytabs imported but HDFS continues writing that error in logs.

Re: Kerberos authentication error with keytab

rrodriguez — Thu, 30 Nov 2017 07:48:30 GMT

Hello @saranvisa

I tested it again after doing a regenerate keytabs and when doing the klist -kt I got the next message.

# klist -kt hdfs.keytab
Keytab name: FILE:hdfs.keytab
klist: Unsupported key table format version number while starting keytab scan

This is not the same for other keytab files in other directories into /var/run/cloudera-scm-agent/process just for some of them.

Any idea of what's happening? Why some processes are getting empty keytab files? I don't understand.

Thank you for the help

Re: Kerberos authentication error with keytab

rrodriguez — Thu, 30 Nov 2017 08:08:39 GMT

Hi again @saranvisa,

I checked the logs and I saw that the error that I was getting on starting a service was caused from a certain process so I got in that directory and looked for the error on hdfs.keytab. When doing the klist -kt hdfs.keytab I got the principals list, tried to make a kinit with one of them and it worked well.

What I've seen is that the imported keytabs I was trying to klist were some old keytab files, modified few weeks ago, and the logs gave me the clue on which directory test the keytab files.

So we are at the same point, seems that krb5-workstation commands work fine, keytabs were generated right and the service keeps outputing the same error again and again.

Some more ideas to test?

Thank you

Re: Kerberos authentication error with keytab

rrodriguez — Wed, 13 Dec 2017 12:00:18 GMT

We surpassed the error just configuring Cloudera to authenticate to a local KDC, we were using a KDC provided by WSO2, this problem got solved but not with the scenario it appeared first.

Re: Kerberos authentication error with keytab

Zubair — Wed, 03 Apr 2019 23:41:23 GMT

Do you wanted to check the proper Authentication is Happenning between Cluster and AD, make sure port,

Re: Kerberos authentication error with keytab

salimhussain — Mon, 12 Oct 2020 12:37:29 GMT

When you get below error message when doing kinit using a keytab file

klist: Unsupported key table format version number while starting keytab scan

Make sure that keytab file is not of zero byte
e.g This is Zero byte keytab file and you will get the above error when trying to do kinit with it

-rw------- 1 cloudera-scm cloudera-scm 0 Aug 30 12:15 ./32-cloudera-mgmt-SERVICEMONITOR/cmon.keytab

A good keytab file will have non-zero size e.g. 778 for the below file

-rw------- 1 cloudera-scm cloudera-scm 778 Oct 12 05:21 ./150-cloudera-mgmt-SERVICEMONITOR/cmon.keytab