Kerberos issue while setting up with Ambari 2.7.5

Question

Failing at 38%

NAMENODE : AMBARI server

stderr: errors-440.txt

stdout: output-440.txt

2021-07-22 09:41:36,191 - Processing identities...
2021-07-22 09:41:36,222 - Creating keytab file for hdpcluster-072221@ on host myserver.com
2021-07-22 09:41:36,230 - Processing identities completed.

DATANODE : HDP CLUSTER

stdout: /var/lib/ambari-agent/data/output-438.txt

2021-07-22 09:41:34,474 - Missing keytabs:
Keytab: /etc/security/keytabs/kerberos.service_check.072221.keytab Principal: hdpcluster-072221

Command completed successfully!

This is the krb5.conf file placed in ambari server (name node) and hdpcluster 3.1.5 (datanode)

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

# output settings
[logging]
default = FILE:/tmp/krb5libs.log
kdc = FILE:/tmp/krb5kdc.log
admin_server = FILE:/tmp/kadmind.log

#Connection default configuration
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}
udp_preference_limit= 1

[realms]
EXAMPLE.COM = {
kdc = myserver.com:88
admin_server = myserver.com
}

# domain to realm relationship (optional)
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

@Shelton

Shelton · Accepted Answer

@ambari275
These are the steps to follow see below

Assumptions
logged as root
clustername=test
REALM= DOMAIN.COM
Hostname = host1

logged in as root

[root@host1]#

Switch to user HDFS the HDFS superuser

[root@host1]# su - hdfs

Check the HDFS associated keytab generated

[hdfs@host1 ~]$ cd /etc/security/keytabs/
[hdfs@host1 keytabs]$ ls

Sample output

atlas.service.keytab hdfs.headless.keytab knox.service.keytab oozie.service.keytab

Now use the hdfs.headless.keytab to get the associated principal

[hdfs@host1 keytabs]$ klist -kt /etc/security/keytabs/hdfs.headless.keytab

Expected output

Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM

Grab a Kerberos ticket by using the keytab+ principal like username/pèassword to authenticate to KDC

[hdfs@host1 keytabs]$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-test@DOMAIN.COM

Check you no have a valid Kerberos ticket

[hdfs@host1 keytabs]$ klist

Sample output

Ticket cache: FILE:/tmp/krb5cc_1013
Default principal: hdfs-test@DOMAIN.COM
Valid starting Expires Service principal
07/26/2021 10:03:17 07/27/2021 10:03:17 krbtgt/DOMAIN.COM@DOMAIN.COM

Now you can list successfully the HDFS directories, remember to -ls it seems you forgot it in your earlier command

[hdfs@host1 keytabs]$ hdfs dfs -ls /
Found 9 items
drwxrwxrwx - yarn hadoop 0 2018-09-24 00:31 /app-logs
drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:22 /apps
drwxr-xr-x - yarn hadoop 0 2018-09-24 00:12 /ats
drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:12 /hdp
drwxr-xr-x - mapred hdfs 0 2018-09-24 00:12 /mapred
drwxrwxrwx - mapred hadoop 0 2018-09-24 00:12 /mr-history
drwxrwxrwx - spark hadoop 0 2021-07-26 10:04 /spark2-history
drwxrwxrwx - hdfs hdfs 0 2021-07-26 00:57 /tmp
drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:23 /user

Voila happy hadooping and remember to accept the best response so other users could reference it

Shelton · Answer

@ambari275

From the onset, I see you left the defaults and I doubt whether that really maps to your cluster.
Here is a list of outputs I need to validate

$ hostname -f [Where you installed the kerberos server]
/etc/hosts
/var/kerberos/krb5kdc/kadm5.acl
/var/kerberos/krb5kdc/kdc.conf

On the Kerberos server can you run

# kadmin.local

Then

list_principals

q to quit

The hostname -f output on the Kerberos server should replace kdc and admin_server in krb5.conf

Here is an example

OS: Centos 7

Cluster Realm

HOTEL.COM

My hosts entry is for a class C network so yours could be different but your host name must be resolved by DNS

[root@test ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.153 test.hotel.com test

[root@test ~]# hostname -f

test.hotel.com

[root@test ~]# cat /var/kerberos/krb5kdc/kadm5.acl

*/admin@HOTEL.COM *

[root@test ~]# cat /etc/krb5.conf

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = HOTEL.COM
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
HOTEL.COM = {
kdc = test.hotel.com
admin_server =test.hotel.com
}
[domain_realm]
.hotel.com = HOTEL.COM
hotel.com = HOTEL.COM

[root@test ~]# cat /var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
HOTEL.COM = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
}
[realms]
HOTEL.COM = {
master_key_type = des-cbc-crc
database_name = /var/kerberos/krb5kdc/principal
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = des-cbc-crc:normal des3-cbc-raw:normal des3-cbc-sha1:norm
al des-cbc-crc:v4 des-cbc-crc:afs3
kadmind_port = 749
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/dict/words
}

Once you share the above then I could figure out where the issue could be.

Happy hadooping

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.