Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kerberos on Quickstart Docker - Generic preauthentication failure

avatar
Explorer

Hello,

 

I am attempting to get the cloudera quickstart (on Docker) to talk to an external Kerberos KDC server (also in Docker, but on the same Docker network) for testing purposes.  The error that has hung me up for a while now is:

 

kinit: Generic preauthentication failure while getting initial credentials

 

On the KDC logging side, it looks something like:

 

 

Jan 23 18:48:04 k-server krb5kdc[23](info): AS_REQ (1 etypes {17}) 192.168.123.2: NEEDED_PREAUTH: HTTP/quickstart.cloudera@REALM.COM for krbtgt/REALM.COM@REALM.COM, Additional pre-authentication required

Jan 23 18:48:04 k-server krb5kdc[23](info): closing down fd 13

Jan 23 18:48:04 k-server krb5kdc[23](info): AS_REQ (1 etypes {17}) 192.168.123.2: NEEDED_PREAUTH: HTTP/quickstart.cloudera@REALM.COM for krbtgt/REALM.COM@REALM.COM, Additional pre-authentication required

Jan 23 18:48:04 k-server krb5kdc[23](info): closing down fd 13

Jan 23 18:49:12 k-server krb5kdc[23](info): AS_REQ (1 etypes {17}) 192.168.123.2: NEEDED_PREAUTH: host/quickstart.cloudera@REALM.COM for krbtgt/REALM.COM@REALM.COM, Additional pre-authentication required

Jan 23 18:49:12 k-server krb5kdc[23](info): closing down fd 13

 

The below reproduces the error at the command line.  The real problem is that this happens within the quickstart Hadoop services for `hdfs/quickstart.cloudera`, `hue/quickstart.cloudera`, etc.

 

[root@quickstart cloudera-scm-server]# kadmin -p user/admin -w password -q "addprinc -randkey host/quickstart.cloudera"

[root@quickstart cloudera-scm-server]# kadmin -p user/admin -w password -q "ktadd -k /etc/krb5.keytab host/quickstart.cloudera"

[root@quickstart cloudera-scm-server]# kinit -k -t /etc/krb5.keytab -V

Using default cache: /tmp/krb5cc_0

Using principal: host/quickstart.cloudera@REALM.COM

Using keytab: /etc/krb5.keytab

kinit: Generic preauthentication failure while getting initial credentials

 

 

Specifically, with regard to other discussions I have seen, I have tried recreating credentials, checking the kvno, and munging with /etc/hosts.  I have even tried starting from scratch a few times.

This is an example error log from the HDFS service:

Tue Jan 23 18:28:04 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Tue Jan 23 18:28:05 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Tue Jan 23 18:28:07 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Tue Jan 23 18:28:11 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.

 

3 REPLIES 3

avatar
Explorer

More logging enabled by setting the environment variable `KRB5_TRACE=/path/to/my/log`

I changed the encryption type to aes128-cts here, and have tried a handful of others:

[12979] 1519323668.876454: Getting initial credentials for host/quickstart.cloudera@REALM.COM
[12979] 1519323668.876719: Looked up etypes in keytab: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, des-hmac-sha1, des, des-cbc-crc
[12979] 1519323668.876744: Sending request (222 bytes) to REALM.COM
[12979] 1519323668.876787: Resolving hostname k-server
[12979] 1519323668.877148: Initiating TCP connection to stream 192.168.123.2:88
[12979] 1519323668.877263: Sending TCP request to stream 192.168.123.2:88
[12979] 1519323668.877579: Received answer from stream 192.168.123.2:88
[12979] 1519323668.877629: Response was not from master KDC
[12979] 1519323668.877647: Received error from KDC: -1765328359/Additional pre-authentication required
[12979] 1519323668.877684: Processing preauth types: 136, 19, 2, 133
[12979] 1519323668.877692: Selected etype info: etype aes128-cts, salt "REALM.COMhostquickstart.cloudera", params ""
[12979] 1519323668.877695: Received cookie: MIT
[12979] 1519323668.877743: Retrieving host/quickstart.cloudera@REALM.COM from FILE:myfile.keytab (vno 0, enctype aes128-cts) with result: -1765328196/Bad encryption type
[12979] 1519323668.877749: Preauth module encrypted_timestamp (2) (flags=1) returned: -1765328196/Bad encryption type
[12979] 1519323668.877752: Produced preauth for next request: 133
[12979] 1519323668.877771: Getting initial credentials for host/quickstart.cloudera@REALM.COM
[12979] 1519323668.877841: Looked up etypes in keytab: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, des-hmac-sha1, des, des-cbc-crc
[12979] 1519323668.877857: Sending request (222 bytes) to REALM.COM (master)


And the keytab itself:

Keytab name: FILE:myfile.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (aes256-cts-hmac-sha1-96) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (aes128-cts-hmac-sha1-96) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (des3-cbc-sha1) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (arcfour-hmac) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (etype 26) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (etype 25) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (des-hmac-sha1) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (des-cbc-md5) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (aes256-cts-hmac-sha1-96) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (aes128-cts-hmac-sha1-96) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (des3-cbc-sha1) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (arcfour-hmac) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (etype 26) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (etype 25) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (des-hmac-sha1) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (des-cbc-md5) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (aes256-cts-hmac-sha1-96) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (aes128-cts-hmac-sha1-96) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (des3-cbc-sha1) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (arcfour-hmac) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (etype 26) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (etype 25) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (des-hmac-sha1) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (des-cbc-md5) 

avatar
Master Guru

@colearendt,

 

I haven't played with the Docker release.  Why are you kiniting from the commandline with the host principal?

CM/CDH don't use the host principal.

Any idea how the myfile.keytab gets created?

 

I guess I am curious what you are trying to do and why when you get the error.

avatar
Explorer

Thanks for the reply!  To be honest, I'm not 100% sure what I'm doing haha.  HDFS is failing to kinit when the service tries to come back up after enabling Kerberos auth.  The above is my attempt to reproduce the error on the command line, since I am not in control when the hdfs service tries to come up.  I forget if I attached this above - this is the service error from the Cloudera Manager when the HDFS service tries to come up:

stdout:

Fri Feb 23 14:52:47 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Fri Feb 23 14:52:49 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Fri Feb 23 14:52:51 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Fri Feb 23 14:52:54 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.


stderr:

+ export HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.6.0-shaded.jar
+ HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.6.0-shaded.jar
+ set -x
+ replace_conf_dir
+ find /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE -type f '!' -path '/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/logs/' '!' -name '.log' '!' -name '*.keytab' '!' -name '*jceks' - 
exec perl -pi -e 's#CMF_CONF_DIR#/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE#g' '{}' ';'
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']'
+ ulimit -l
+ export HADOOP_IDENT_STRING=hdfs
+ HADOOP_IDENT_STRING=hdfs
+ '[' -n '' ']'
+ acquire_kerberos_tgt hdfs.keytab
+ '[' -z hdfs.keytab ']'
+ '[' -n hdfs/quickstart.cloudera@REALM.COM ']'
+ '[' -d /usr/kerberos/bin ']'
+ which kinit
+ '[' 0 -ne 0 ']'
++ id -u
+ export KRB5CCNAME=/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497
+ KRB5CCNAME=/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497
+ echo 'using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal'
+ echo 'using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache'
+ kinit -c /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 -kt /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/hdfs.keytab hdfs/quickstart.cloudera@REALM.COM
kinit: Generic preauthentication failure while getting initial credentials
+ '[' 1 -ne 0 ']'
+ echo 'kinit was not successful.'
+ exit 1

For the record, I get the same error that I got above if I try to manually use the keytab in the `29-hdfs-NAMENODE` folder.

Further, the reason that I added the `host` principal was because authenticating manually without it said it failed on needing the host principal.  So clearly my "reproduction" is not accurate, but I arrived at the same error (and an error that I do not encounter on other hosts), so I was hopeful to be onto something.  No luck so far though.

The "Bad encryption type" has completely stymied me.  I have tried a bunch of the other available encryption types to no avail.  (Oh and myfile.keytab was created by me with the commands above since it was my attempt at a repro.  But as I mentioned, using the Cloudera-generated keytab also results in the same issue).