Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Kerberos on Quickstart Docker - Generic preauthentication failure

avatar
author-rank colearendt
Explorer

Created on ‎01-23-2018 11:00 AM - edited ‎09-16-2022 05:46 AM

Hello,

 

I am attempting to get the cloudera quickstart (on Docker) to talk to an external Kerberos KDC server (also in Docker, but on the same Docker network) for testing purposes.  The error that has hung me up for a while now is:

 

kinit: Generic preauthentication failure while getting initial credentials

 

On the KDC logging side, it looks something like:

 

 

Jan 23 18:48:04 k-server krb5kdc[23](info): AS_REQ (1 etypes {17}) 192.168.123.2: NEEDED_PREAUTH: HTTP/quickstart.cloudera@REALM.COM for krbtgt/REALM.COM@REALM.COM, Additional pre-authentication required

Jan 23 18:48:04 k-server krb5kdc[23](info): closing down fd 13

Jan 23 18:48:04 k-server krb5kdc[23](info): AS_REQ (1 etypes {17}) 192.168.123.2: NEEDED_PREAUTH: HTTP/quickstart.cloudera@REALM.COM for krbtgt/REALM.COM@REALM.COM, Additional pre-authentication required

Jan 23 18:48:04 k-server krb5kdc[23](info): closing down fd 13

Jan 23 18:49:12 k-server krb5kdc[23](info): AS_REQ (1 etypes {17}) 192.168.123.2: NEEDED_PREAUTH: host/quickstart.cloudera@REALM.COM for krbtgt/REALM.COM@REALM.COM, Additional pre-authentication required

Jan 23 18:49:12 k-server krb5kdc[23](info): closing down fd 13

 

The below reproduces the error at the command line.  The real problem is that this happens within the quickstart Hadoop services for `hdfs/quickstart.cloudera`, `hue/quickstart.cloudera`, etc.

 

[root@quickstart cloudera-scm-server]# kadmin -p user/admin -w password -q "addprinc -randkey host/quickstart.cloudera"

[root@quickstart cloudera-scm-server]# kadmin -p user/admin -w password -q "ktadd -k /etc/krb5.keytab host/quickstart.cloudera"

[root@quickstart cloudera-scm-server]# kinit -k -t /etc/krb5.keytab -V

Using default cache: /tmp/krb5cc_0

Using principal: host/quickstart.cloudera@REALM.COM

Using keytab: /etc/krb5.keytab

kinit: Generic preauthentication failure while getting initial credentials

 

 

Specifically, with regard to other discussions I have seen, I have tried recreating credentials, checking the kvno, and munging with /etc/hosts.  I have even tried starting from scratch a few times.

This is an example error log from the HDFS service:

Tue Jan 23 18:28:04 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Tue Jan 23 18:28:05 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Tue Jan 23 18:28:07 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Tue Jan 23 18:28:11 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/12-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.

 

4,517 Views
0 Kudos
3 REPLIES 3

avatar
author-rank colearendt
Explorer

Created ‎02-22-2018 10:30 AM

More logging enabled by setting the environment variable `KRB5_TRACE=/path/to/my/log`

I changed the encryption type to aes128-cts here, and have tried a handful of others:

[12979] 1519323668.876454: Getting initial credentials for host/quickstart.cloudera@REALM.COM
[12979] 1519323668.876719: Looked up etypes in keytab: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, des-hmac-sha1, des, des-cbc-crc
[12979] 1519323668.876744: Sending request (222 bytes) to REALM.COM
[12979] 1519323668.876787: Resolving hostname k-server
[12979] 1519323668.877148: Initiating TCP connection to stream 192.168.123.2:88
[12979] 1519323668.877263: Sending TCP request to stream 192.168.123.2:88
[12979] 1519323668.877579: Received answer from stream 192.168.123.2:88
[12979] 1519323668.877629: Response was not from master KDC
[12979] 1519323668.877647: Received error from KDC: -1765328359/Additional pre-authentication required
[12979] 1519323668.877684: Processing preauth types: 136, 19, 2, 133
[12979] 1519323668.877692: Selected etype info: etype aes128-cts, salt "REALM.COMhostquickstart.cloudera", params ""
[12979] 1519323668.877695: Received cookie: MIT
[12979] 1519323668.877743: Retrieving host/quickstart.cloudera@REALM.COM from FILE:myfile.keytab (vno 0, enctype aes128-cts) with result: -1765328196/Bad encryption type
[12979] 1519323668.877749: Preauth module encrypted_timestamp (2) (flags=1) returned: -1765328196/Bad encryption type
[12979] 1519323668.877752: Produced preauth for next request: 133
[12979] 1519323668.877771: Getting initial credentials for host/quickstart.cloudera@REALM.COM
[12979] 1519323668.877841: Looked up etypes in keytab: aes256-cts, aes128-cts, des3-cbc-sha1, rc4-hmac, des-hmac-sha1, des, des-cbc-crc
[12979] 1519323668.877857: Sending request (222 bytes) to REALM.COM (master)


And the keytab itself:

Keytab name: FILE:myfile.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (aes256-cts-hmac-sha1-96) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (aes128-cts-hmac-sha1-96) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (des3-cbc-sha1) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (arcfour-hmac) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (etype 26) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (etype 25) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (des-hmac-sha1) 
   1 02/22/18 18:20:16 hdfs/quickstart.cloudera@REALM.COM (des-cbc-md5) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (aes256-cts-hmac-sha1-96) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (aes128-cts-hmac-sha1-96) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (des3-cbc-sha1) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (arcfour-hmac) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (etype 26) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (etype 25) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (des-hmac-sha1) 
   1 02/22/18 18:20:24 HTTP/quickstart.cloudera@REALM.COM (des-cbc-md5) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (aes256-cts-hmac-sha1-96) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (aes128-cts-hmac-sha1-96) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (des3-cbc-sha1) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (arcfour-hmac) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (etype 26) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (etype 25) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (des-hmac-sha1) 
   1 02/22/18 18:20:29 host/quickstart.cloudera@REALM.COM (des-cbc-md5)
4,430 Views
0 Kudos

avatar
author-rank bgooley author-rank
Master Guru

Created ‎02-22-2018 11:04 AM

@colearendt,

 

I haven't played with the Docker release.  Why are you kiniting from the commandline with the host principal?

CM/CDH don't use the host principal.

Any idea how the myfile.keytab gets created?

 

I guess I am curious what you are trying to do and why when you get the error.

4,425 Views
0 Kudos

avatar
author-rank colearendt
Explorer

Created on ‎02-23-2018 07:00 AM - edited ‎02-23-2018 07:02 AM

Thanks for the reply!  To be honest, I'm not 100% sure what I'm doing haha.  HDFS is failing to kinit when the service tries to come back up after enabling Kerberos auth.  The above is my attempt to reproduce the error on the command line, since I am not in control when the hdfs service tries to come up.  I forget if I attached this above - this is the service error from the Cloudera Manager when the HDFS service tries to come up:

stdout:

Fri Feb 23 14:52:47 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Fri Feb 23 14:52:49 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Fri Feb 23 14:52:51 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.
Fri Feb 23 14:52:54 UTC 2018
JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME
using 5 as CDH_VERSION
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR
using  as SECURE_USER
using  as SECURE_GROUP
unlimited
/usr/bin/kinit
using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal
using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache
kinit was not successful.


stderr:

+ export HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.6.0-shaded.jar
+ HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.6.0-shaded.jar
+ set -x
+ replace_conf_dir
+ find /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE -type f '!' -path '/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/logs/' '!' -name '.log' '!' -name '*.keytab' '!' -name '*jceks' - 
exec perl -pi -e 's#CMF_CONF_DIR#/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE#g' '{}' ';'
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']'
+ ulimit -l
+ export HADOOP_IDENT_STRING=hdfs
+ HADOOP_IDENT_STRING=hdfs
+ '[' -n '' ']'
+ acquire_kerberos_tgt hdfs.keytab
+ '[' -z hdfs.keytab ']'
+ '[' -n hdfs/quickstart.cloudera@REALM.COM ']'
+ '[' -d /usr/kerberos/bin ']'
+ which kinit
+ '[' 0 -ne 0 ']'
++ id -u
+ export KRB5CCNAME=/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497
+ KRB5CCNAME=/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497
+ echo 'using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal'
+ echo 'using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache'
+ kinit -c /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 -kt /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/hdfs.keytab hdfs/quickstart.cloudera@REALM.COM
kinit: Generic preauthentication failure while getting initial credentials
+ '[' 1 -ne 0 ']'
+ echo 'kinit was not successful.'
+ exit 1

For the record, I get the same error that I got above if I try to manually use the keytab in the `29-hdfs-NAMENODE` folder.

Further, the reason that I added the `host` principal was because authenticating manually without it said it failed on needing the host principal.  So clearly my "reproduction" is not accurate, but I arrived at the same error (and an error that I do not encounter on other hosts), so I was hopeful to be onto something.  No luck so far though.

The "Bad encryption type" has completely stymied me.  I have tried a bunch of the other available encryption types to no avail.  (Oh and myfile.keytab was created by me with the commands above since it was my attempt at a repro.  But as I mentioned, using the Cloudera-generated keytab also results in the same issue).

4,411 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements