Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

Hue Kerberos error: "TICKET NOT RENEWABLE"

avatar
author-rank geko
Guru

Created on ‎02-24-2014 07:14 AM - edited ‎09-16-2022 01:54 AM

Hi,

I am currently in the process of enabling security in our cluster (CDH4.5, CM4.8) according the documentation here => http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Configuring-Hadoop-Secur...

 

Everything went fine until step 14, starting all the services. The service "Kerberos Ticket Renewer" doesn't start, the latest log entries are:

""

[24/Feb/2014 15:41:39 +0000] settings     INFO     Welcome to Hue 2.5.0
[24/Feb/2014 15:41:40 +0000] kt_renewer   INFO     Reinitting kerberos from keytab: /usr/bin/kinit -k -t /var/run/cloudera-scm-agent/process/1715-hue-KT_RENEWER/hue.keytab -c /tmp/hue_krb5_ccache hue/hadoop-pg-1.cluster
[24/Feb/2014 15:41:42 +0000] kt_renewer   INFO     Renewing kerberos ticket to work around kerberos 1.8.1: /usr/bin/kinit -R -c /tmp/hue_krb5_ccache
[24/Feb/2014 15:41:42 +0000] kt_renewer   ERROR    Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/hadoop-pg-1.cluster' is still renewable:
  $ kinit -f -c /tmp/hue_krb5_ccache
If the 'renew until' date is the same as the 'valid starting' date, the ticket cannot be renewed. Please check your KDC configuration, and the ticket renewal policy (maxrenewlife) for the 'hue/hadoop-pg-1.cluster' and `krbtgt' principals.

""

 

The logs of the KDC shows:

""

Feb 24 15:41:33 hadoop-pg-1 krb5kdc[4475](info): AS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: NEEDED_PREAUTH: hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG, Additional pre-authentication required
Feb 24 15:41:33 hadoop-pg-1 krb5kdc[4475](info): AS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: ISSUE: authtime 1393252893, etypes {rep=18 tkt=18 ses=18}, hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG
Feb 24 15:41:35 hadoop-pg-1 krb5kdc[4475](info): TGS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: TICKET NOT RENEWABLE: authtime 0, hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG, KDC can't fulfill requested option
Feb 24 15:41:35 hadoop-pg-1 krb5kdc[4475](info): TGS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: TICKET NOT RENEWABLE: authtime 0, hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG, KDC can't fulfill requested option

""

 

The KDC config looks like:

""

[kdcdefaults]
kdc_ports = 750,88

[realms]
HADOOP-PG = {
database_name = /var/lib/krb5kdc/principal
admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
acl_file = /etc/krb5kdc/kadm5.acl
key_stash_file = /etc/krb5kdc/stash
kdc_ports = 750,88
max_life = 1d 0h 0m 0s
max_renewable_life = 90d 0h 0m 0s
master_key_type = des3-hmac-sha1
supported_enctypes = aes256-cts:normal arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3
default_principal_flags = +preauth +renewable
}

""

 

Additionally I set the following:

""

kadmin.local: modprinc -maxlife "1 day" -maxrenewlife "90 day" +allow_renewable hue/hadoop-pg-1.cluster@HADOOP-PG

""

 

Some hints, where to investigate to resolve this issue?

 

br, Gerd

35,805 Views
0 Kudos
1
1 ACCEPTED SOLUTION

avatar
author-rank Grizzly author-rank
Master Collaborator

Created ‎02-24-2014 08:22 AM

Consider the following examples:

 

First the /etc/krb5.conf  In this example a second domain is configured (Active Directory) for cross realm authentication with AES256 encryption being used by AD.  Using AES256 means that one must install the JCE Policy Files For JDK6 or the JCE Policy Files for JDK7 to use stron encryption like AES256. Note the Items in bold that are pointed, out, they should be set in that specific file (krb5.condif)

 

[logging]
   default = FILE:/var/log/krb5libs.log
   kdc = FILE:/var/log/krb5kdc.log
   admin_server = FILE:/var/log/kadmind.log
[libdefaults]
   default_realm = TEST.LAB
   dns_lookup_realm = false
   dns_lookup_kdc = false
   ticket_lifetime = 24h
   renew_lifetime = 7d
   forwardable = true
[realms]
TEST.ORG.LAB = {
   kdc = Win2k8x64-AD4.test.org.lab:88
   kdc = Win2k8x64-AD2.test.org.lab:88
   admin_server = Win2k8x64-AD4.test.org.lab:749
   admin_server = Win2k8x64-AD2.test.org.lab:749
   default_domain = test.org.lab
}
TEST.LAB = {
   kdc = kdc1.test.lab:88
   admin_server = kdc1.test.lab:749
   default_domain = test.lab
}
[domain_realm]
   .test.lab = TEST.LAB
   test.lab = TEST.LAB
   .test.org.lab = TEST.ORG.LAB
   test.org.lab = TEST.ORG.LAB

Consider the following for the /var/kerberose/krb5kdc/kdc.conf, calling out items to set in this file as Bold Text,  below.

 

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
  TEST.LAB = {
  #master_key_type = aes256-cts
  max_renewable_life = 7d 0h 0m 0s
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  default_principal_flags = +renewable
 }

 

View solution in original post

35,599 Views
7 REPLIES 7

avatar
author-rank Grizzly author-rank
Master Collaborator

Created ‎02-24-2014 08:22 AM

Consider the following examples:

 

First the /etc/krb5.conf  In this example a second domain is configured (Active Directory) for cross realm authentication with AES256 encryption being used by AD.  Using AES256 means that one must install the JCE Policy Files For JDK6 or the JCE Policy Files for JDK7 to use stron encryption like AES256. Note the Items in bold that are pointed, out, they should be set in that specific file (krb5.condif)

 

[logging]
   default = FILE:/var/log/krb5libs.log
   kdc = FILE:/var/log/krb5kdc.log
   admin_server = FILE:/var/log/kadmind.log
[libdefaults]
   default_realm = TEST.LAB
   dns_lookup_realm = false
   dns_lookup_kdc = false
   ticket_lifetime = 24h
   renew_lifetime = 7d
   forwardable = true
[realms]
TEST.ORG.LAB = {
   kdc = Win2k8x64-AD4.test.org.lab:88
   kdc = Win2k8x64-AD2.test.org.lab:88
   admin_server = Win2k8x64-AD4.test.org.lab:749
   admin_server = Win2k8x64-AD2.test.org.lab:749
   default_domain = test.org.lab
}
TEST.LAB = {
   kdc = kdc1.test.lab:88
   admin_server = kdc1.test.lab:749
   default_domain = test.lab
}
[domain_realm]
   .test.lab = TEST.LAB
   test.lab = TEST.LAB
   .test.org.lab = TEST.ORG.LAB
   test.org.lab = TEST.ORG.LAB

Consider the following for the /var/kerberose/krb5kdc/kdc.conf, calling out items to set in this file as Bold Text,  below.

 

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
  TEST.LAB = {
  #master_key_type = aes256-cts
  max_renewable_life = 7d 0h 0m 0s
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  default_principal_flags = +renewable
 }

 

35,600 Views

avatar
author-rank geko
Guru

Created ‎02-25-2014 12:26 AM

Hi Tgrayson,

 

thanks for your answer. Seems like adding the {ticket_|renew_}lifetime parameters solved the problem. After inserting them, reducing the original renew-lifetime to 7d and restarting all the services it looks good and I can proceed with the doc mentioned in the initial post.

 

thanks, Gerd

35,575 Views
0 Kudos

avatar
author-rank narayanab16
Explorer

Created on ‎08-18-2014 09:58 AM - edited ‎08-18-2014 10:43 AM

Hi,

Im using only kerberos not AD

I get an error please let me know why?

Im trying to run job from sqoop sql to hdfs

http://cloudera-cdh05.narayana.local:8888/sqoop/#jobs

on browser

Error: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

[root@xeon narayana]# vim /var/kerberos/krb5kdc/kdc.conf
-----------------------------

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 NARAYANA.LOCAL = {
  #master_key_type = aes256-cts
  # Narayana added
  max_renewable_life = 10000d
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  # Narayana added
  default_principal_flags = +renewable
}

 

Clients
----------

[cloudera@cloudera-cdh05 hue]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: cloudera/cloudera-cdh05.narayana.local@NARAYANA.LOCAL

Valid starting     Expires            Service principal
08/18/14 22:24:54  08/19/14 22:24:54  krbtgt/NARAYANA.LOCAL@NARAYANA.LOCAL
    renew until 08/18/14 22:24:54

[cloudera@cloudera-cdh05 hue]$ cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = NARAYANA.LOCAL
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 10000h
 renew_lifetime = 10000d
 forwardable = true
 udp_preference_limit = 1

[realms]
 NARAYANA.LOCAL = {
  kdc = 192.168.1.10
  admin_server = 192.168.1.10
 }

[domain_realm]
 .narayana.local = NARAYANA.LOCAL
 narayana.local = NARAYANA.LOCAL

 

Error
-------
/var/log/hue/error.log
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:11 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:41 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response

35,175 Views
0 Kudos

avatar
author-rank Grizzly author-rank
Master Collaborator

Created ‎08-18-2014 05:56 PM

We provide example kdc.conf and krb5.conf here:

 

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Security-Guide/cdh5s...

 

if you us "kadmin.local" and then the "getprinc" command to specifically examine principals in question, it will tell you what properties are set upon creation of the principal.

 

Your krb5.conf needs to provide better ticket and renew lifetime settings.  See the example for guidance.

 

Todd

35,164 Views
0 Kudos

avatar
author-rank narayanab16
Explorer

Created on ‎08-18-2014 09:35 PM - edited ‎08-18-2014 09:50 PM

Hi Todd,

 

What am i missing i really don't understand, the error remains the same.

 

I dont see anything wrong with configuration, i update conf as decribed in the troubleshoot.

Below is my conf list, mapreduce jobs are working as expected

 

Server

[root@xeon narayana]# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 NARAYANA.LOCAL = {
  #master_key_type = aes256-cts
  # Narayana added
  max_renewable_life = 7d 0h 0m 0s
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  # Narayana added
  default_principal_flags = +renewable
}

clients
---------

 

[cloudera@cloudera-cdh05 hue]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: cloudera/cloudera-cdh05.narayana.local@NARAYANA.LOCAL

Valid starting     Expires            Service principal
08/19/14 09:48:35  08/20/14 09:48:33  krbtgt/NARAYANA.LOCAL@NARAYANA.LOCAL
    renew until 08/19/14 09:48:35
[cloudera@cloudera-cdh05 hue]$ cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = NARAYANA.LOCAL
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 #udp_preference_limit = 1

[realms]
 NARAYANA.LOCAL = {
  kdc = 192.168.1.10
  admin_server = 192.168.1.10
  default_domain = narayana.local
 }

[domain_realm]
 .narayana.local = NARAYANA.LOCAL
 narayana.local = NARAYANA.LOCAL

[root@cloudera-cdh06 ~]# cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = NARAYANA.LOCAL
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 #udp_preference_limit = 1

[realms]
 NARAYANA.LOCAL = {
  kdc = 192.168.1.10
  admin_server = 192.168.1.10
  default_domain = narayana.local
 }

[domain_realm]
 .narayana.local = NARAYANA.LOCAL
 narayana.local = NARAYANA.LOCAL

35,160 Views
0 Kudos

avatar
author-rank Grizzly author-rank
Master Collaborator

Created ‎08-19-2014 11:07 AM

these are false-positive error messages they can be ignored:

[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:11 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:41 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response

 

What is the actual error (other than that log message that can be ignored), are you not able to access/use services?

 

did  you examine how the principals were created with the getprinc subcommand of kadmin.local?

35,150 Views
0 Kudos

avatar
author-rank Parwinderjit
New Contributor

Created on ‎12-09-2018 09:13 PM - edited ‎12-09-2018 09:20 PM

 

Can somebody please help ? 

 

kinit: Failed to store credentials: Internal credentials cache error (filename: /hue_krb5_ccache) while getting initial credentials

 

[09/Dec/2018 21:06:24 -0800] kt_renewer   INFO     Reinitting kerberos retry attempt 2 from keytab /bin/kinit -k -t /run/cloudera-scm-agent/process/450-hue-KT_RENEWER/hue.keytab -c /hue_krb5_ccache hue/kabo1.unraveldatalab.com@unravel.COM

[09/Dec/2018 21:06:24 -0800] kt_renewer   ERROR    Couldn't reinit from keytab! `kinit' exited with 1.

 

kinit: Failed to store credentials: Internal credentials cache error (filename: /hue_krb5_ccache) while getting initial credentials

 

[09/Dec/2018 21:06:24 -0800] kt_renewer   ERROR    FATAL: max_retries of 3 reached. Exiting...

[09/Dec/2018 21:06:28 ] settings     INFO     Welcome to Hue 3.9.0

[09/Dec/2018 21:06:31 -0800] __init__     INFO     Couldn't import snappy. Support for snappy compression disabled.

[09/Dec/2018 21:06:31 -0800] kt_renewer   INFO     Reinitting kerberos retry attempt 0 from keytab /bin/kinit -k -t /run/cloudera-scm-agent/process/450-hue-KT_RENEWER/hue.keytab -c /hue_krb5_ccache hue/kabo1.unraveldatalab.com@unravel.COM

[09/Dec/2018 21:06:31 -0800] kt_renewer   ERROR    Couldn't reinit from keytab! `kinit' exited with 1.

 

kinit: Failed to store credentials: Internal credentials cache error (filename: /hue_krb5_ccache) while getting initial credentials

 

[09/Dec/2018 21:06:34 -0800] kt_renewer   INFO     Reinitting kerberos retry attempt 1 from keytab /bin/kinit -k -t /run/cloudera-scm-agent/process/450-hue-KT_RENEWER/hue.keytab -c /hue_krb5_ccache hue/kabo1.unraveldatalab.com@unravel.COM

[09/Dec/2018 21:06:34 -0800] kt_renewer   ERROR    Couldn't reinit from keytab! `kinit' exited with 1.

 

kinit: Failed to store credentials: Internal credentials cache error (filename: /hue_krb5_ccache) while getting initial credentials

 

[09/Dec/2018 21:06:37 -0800] kt_renewer   INFO     Reinitting kerberos retry attempt 2 from keytab /bin/kinit -k -t /run/cloudera-scm-agent/process/450-hue-KT_RENEWER/hue.keytab -c /hue_krb5_ccache hue/kabo1.unraveldatalab.com@unravel.COM

[09/Dec/2018 21:06:37 -0800] kt_renewer   ERROR    Couldn't reinit from keytab! `kinit' exited with 1.

 

kinit: Failed to store credentials: Internal credentials cache error (filename: /hue_krb5_ccache) while getting initial credentials

 

[09/Dec/2018 21:06:37 -0800] kt_renewer   ERROR    FATAL: max_retries of 3 reached. Exiting...

\

 

 

 

 

[root@kabo1 ~]# cat /etc/krb5.conf

[libdefaults]

default_realm = unravel.COM

dns_lookup_kdc = false

dns_lookup_realm = false

ticket_lifetime = 24h

renew_lifetime = 7d

forwardable = true

default_tkt_enctypes = aes256-cts-hmac-sha1-96

default_tgs_enctypes = aes256-cts-hmac-sha1-96

permitted_enctypes = aes256-cts-hmac-sha1-96

allow_weak_crypto = true

udp_preference_limit = 1

kdc_timeout = 3000

[realms]

unravel.COM = {

kdc = kabo1.unraveldatalab.com

admin_server = kabo1.unraveldatalab.com

}

[domain_realm]

[root@kabo1 ~]# cat  /var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]

kdc_ports = 88

kdc_tcp_ports = 88

 

[realms]

EXAMPLE.COM = {

  #master_key_type = aes256-cts

  max_renewable_life = 7d 0h 0m 0s

  acl_file = /var/kerberos/krb5kdc/kadm5.acl

  dict_file = /usr/share/dict/words

  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab

  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal

  default_principal_flags = +renewable

}

 

 

 

17,875 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements