question Re: Hue Kerberos error: "TICKET NOT RENEWABLE" in Support Questions

Hue Kerberos error: "TICKET NOT RENEWABLE"

geko — Fri, 16 Sep 2022 08:54:33 GMT

Hi,

I am currently in the process of enabling security in our cluster (CDH4.5, CM4.8) according the documentation here => http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.4/Configuring-Hadoop-Security-with-Cloudera-Manager/Configuring-Hadoop-Security-with-Cloudera-Manager.html

Everything went fine until step 14, starting all the services. The service "Kerberos Ticket Renewer" doesn't start, the latest log entries are:

[24/Feb/2014 15:41:39 +0000] settings     INFO     Welcome to Hue 2.5.0
[24/Feb/2014 15:41:40 +0000] kt_renewer   INFO     Reinitting kerberos from keytab: /usr/bin/kinit -k -t /var/run/cloudera-scm-agent/process/1715-hue-KT_RENEWER/hue.keytab -c /tmp/hue_krb5_ccache hue/hadoop-pg-1.cluster
[24/Feb/2014 15:41:42 +0000] kt_renewer   INFO     Renewing kerberos ticket to work around kerberos 1.8.1: /usr/bin/kinit -R -c /tmp/hue_krb5_ccache
[24/Feb/2014 15:41:42 +0000] kt_renewer   ERROR    Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/hadoop-pg-1.cluster' is still renewable:
  $ kinit -f -c /tmp/hue_krb5_ccache
If the 'renew until' date is the same as the 'valid starting' date, the ticket cannot be renewed. Please check your KDC configuration, and the ticket renewal policy (maxrenewlife) for the 'hue/hadoop-pg-1.cluster' and `krbtgt' principals.

The logs of the KDC shows:

Feb 24 15:41:33 hadoop-pg-1 krb5kdc[4475](info): AS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: NEEDED_PREAUTH: hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG, Additional pre-authentication required
Feb 24 15:41:33 hadoop-pg-1 krb5kdc[4475](info): AS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: ISSUE: authtime 1393252893, etypes {rep=18 tkt=18 ses=18}, hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG
Feb 24 15:41:35 hadoop-pg-1 krb5kdc[4475](info): TGS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: TICKET NOT RENEWABLE: authtime 0, hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG, KDC can't fulfill requested option
Feb 24 15:41:35 hadoop-pg-1 krb5kdc[4475](info): TGS_REQ (4 etypes {18 17 16 23}) 10.147.210.1: TICKET NOT RENEWABLE: authtime 0, hue/hadoop-pg-1.cluster@HADOOP-PG for krbtgt/HADOOP-PG@HADOOP-PG, KDC can't fulfill requested option

The KDC config looks like:

[kdcdefaults]
kdc_ports = 750,88

[realms]
HADOOP-PG = {
database_name = /var/lib/krb5kdc/principal
admin_keytab = FILE:/etc/krb5kdc/kadm5.keytab
acl_file = /etc/krb5kdc/kadm5.acl
key_stash_file = /etc/krb5kdc/stash
kdc_ports = 750,88
max_life = 1d 0h 0m 0s
max_renewable_life = 90d 0h 0m 0s
master_key_type = des3-hmac-sha1
supported_enctypes = aes256-cts:normal arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal des:normal des:v4 des:norealm des:onlyrealm des:afs3
default_principal_flags = +preauth +renewable
}

Additionally I set the following:

kadmin.local: modprinc -maxlife "1 day" -maxrenewlife "90 day" +allow_renewable hue/hadoop-pg-1.cluster@HADOOP-PG

Some hints, where to investigate to resolve this issue?

br, Gerd

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

Grizzly — Mon, 24 Feb 2014 16:22:57 GMT

Consider the following examples:

First the /etc/krb5.conf In this example a second domain is configured (Active Directory) for cross realm authentication with AES256 encryption being used by AD. Using AES256 means that one must install the JCE Policy Files For JDK6 or the JCE Policy Files for JDK7 to use stron encryption like AES256. Note the Items in bold that are pointed, out, they should be set in that specific file (krb5.condif)

[logging]
   default = FILE:/var/log/krb5libs.log
   kdc = FILE:/var/log/krb5kdc.log
   admin_server = FILE:/var/log/kadmind.log
[libdefaults]
   default_realm = TEST.LAB
   dns_lookup_realm = false
   dns_lookup_kdc = false
   ticket_lifetime = 24h
   renew_lifetime = 7d
   forwardable = true
[realms]
TEST.ORG.LAB = {
   kdc = Win2k8x64-AD4.test.org.lab:88
   kdc = Win2k8x64-AD2.test.org.lab:88
   admin_server = Win2k8x64-AD4.test.org.lab:749
   admin_server = Win2k8x64-AD2.test.org.lab:749
   default_domain = test.org.lab
}
TEST.LAB = {
   kdc = kdc1.test.lab:88
   admin_server = kdc1.test.lab:749
   default_domain = test.lab
}
[domain_realm]
   .test.lab = TEST.LAB
   test.lab = TEST.LAB
   .test.org.lab = TEST.ORG.LAB
   test.org.lab = TEST.ORG.LAB

Consider the following for the /var/kerberose/krb5kdc/kdc.conf, calling out items to set in this file as Bold Text, below.

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
  TEST.LAB = {
  #master_key_type = aes256-cts
  max_renewable_life = 7d 0h 0m 0s
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  default_principal_flags = +renewable
 }

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

geko — Tue, 25 Feb 2014 08:26:08 GMT

Hi Tgrayson,

thanks for your answer. Seems like adding the {ticket_|renew_}lifetime parameters solved the problem. After inserting them, reducing the original renew-lifetime to 7d and restarting all the services it looks good and I can proceed with the doc mentioned in the initial post.

thanks, Gerd

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

narayanab16 — Mon, 18 Aug 2014 17:43:30 GMT

Hi,

Im using only kerberos not AD

I get an error please let me know why?

Im trying to run job from sqoop sql to hdfs

http://cloudera-cdh05.narayana.local:8888/sqoop/#jobs

on browser

Error: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

[root@xeon narayana]# vim /var/kerberos/krb5kdc/kdc.conf
-----------------------------

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
NARAYANA.LOCAL = {
#master_key_type = aes256-cts
# Narayana added
max_renewable_life = 10000d
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
# Narayana added
default_principal_flags = +renewable
}

Clients
----------

[cloudera@cloudera-cdh05 hue]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: cloudera/cloudera-cdh05.narayana.local@NARAYANA.LOCAL

Valid starting     Expires            Service principal
08/18/14 22:24:54 08/19/14 22:24:54 krbtgt/NARAYANA.LOCAL@NARAYANA.LOCAL
    renew until 08/18/14 22:24:54

[cloudera@cloudera-cdh05 hue]$ cat /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = NARAYANA.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 10000h
renew_lifetime = 10000d
forwardable = true
udp_preference_limit = 1

[realms]
NARAYANA.LOCAL = {
kdc = 192.168.1.10
admin_server = 192.168.1.10
}

[domain_realm]
.narayana.local = NARAYANA.LOCAL
narayana.local = NARAYANA.LOCAL

Error
-------
/var/log/hue/error.log
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:11 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:41 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

Grizzly — Tue, 19 Aug 2014 00:56:43 GMT

We provide example kdc.conf and krb5.conf here:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Security-Guide/cdh5sg_troubleshooting.html

if you us "kadmin.local" and then the "getprinc" command to specifically examine principals in question, it will tell you what properties are set upon creation of the principal.

Your krb5.conf needs to provide better ticket and renew lifetime settings. See the example for guidance.

Todd

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

narayanab16 — Tue, 19 Aug 2014 04:50:30 GMT

Hi Todd,

What am i missing i really don't understand, the error remains the same.

I dont see anything wrong with configuration, i update conf as decribed in the troubleshoot.

Below is my conf list, mapreduce jobs are working as expected

Server

[root@xeon narayana]# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
NARAYANA.LOCAL = {
#master_key_type = aes256-cts
# Narayana added
max_renewable_life = 7d 0h 0m 0s
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
# Narayana added
default_principal_flags = +renewable
}

clients
---------

[cloudera@cloudera-cdh05 hue]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: cloudera/cloudera-cdh05.narayana.local@NARAYANA.LOCAL

Valid starting Expires Service principal
08/19/14 09:48:35 08/20/14 09:48:33 krbtgt/NARAYANA.LOCAL@NARAYANA.LOCAL
renew until 08/19/14 09:48:35
[cloudera@cloudera-cdh05 hue]$ cat /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = NARAYANA.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
#udp_preference_limit = 1

[realms]
NARAYANA.LOCAL = {
kdc = 192.168.1.10
admin_server = 192.168.1.10
default_domain = narayana.local
}

[domain_realm]
.narayana.local = NARAYANA.LOCAL
narayana.local = NARAYANA.LOCAL

[root@cloudera-cdh06 ~]# cat /etc/krb5.conf
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = NARAYANA.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
#udp_preference_limit = 1

[realms]
NARAYANA.LOCAL = {
kdc = 192.168.1.10
admin_server = 192.168.1.10
default_domain = narayana.local
}

[domain_realm]
.narayana.local = NARAYANA.LOCAL
narayana.local = NARAYANA.LOCAL

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

Grizzly — Tue, 19 Aug 2014 18:07:32 GMT

these are false-positive error messages they can be ignored:

[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:08 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:11 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response
[18/Aug/2014 09:57:41 -0700] kerberos_    ERROR    handle_other(): Mutual authentication unavailable on 200 response

What is the actual error (other than that log message that can be ignored), are you not able to access/use services?

did you examine how the principals were created with the getprinc subcommand of kadmin.local?

Re: Hue Kerberos error: "TICKET NOT RENEWABLE"

Parwinderjit — Mon, 10 Dec 2018 05:20:45 GMT

Can somebody please help ?