Support Questions

Find answers, ask questions, and share your expertise

HDP Kerberos enable through Ambari

avatar
Contributor

Hi I have 6 node cluster setup as HDP 2.5 and ambari 2.4

Need to enable Kerberos Security:

Steps Did In ambari server :

1.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/_enabling_kerberos_secu...

2. Yum install Kerberos and clients in all nodes

3. check /etc/krb.conf and checked on all nodes same files and realms

4. checked acl.file and edited and saved and restarted Kdc again

5. Enabling kerberos in ambari it was throwing error in Test Clients:

Error message: An internal system exception occurred: The 'krb5-conf' configuration is not available

500 status codereceived on POST method for API: /api/v1/clusters/Cluster/requests

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Sam Red


On the KDC server which can be the same as the Ambari server or any server in the cluster install both rpm's on all

# yum install -y krb5-server krb5-workstation  

Edit Server Configuration Files

cd  /var/kerberos/krb5kdc 

Edit these 2 files

kdc.conf
kadm5.acl 

Mine are save

# cat kadm5.acl 
*/admin@TEST.COM      * 

# cat kdc.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
[realms]
 TEST.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal 

Edit the Client Configuration

This file should be available on the KDC server edit it and copy it to the same path to all the other nodes

# cat /etc/krb5.conf

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = TEST.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  test.com = TEST.COM
  .test.com = TEST.COM
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  TEST.COM = {
    admin_server = kdc.TEST.com
    kdc = kdc.test.com
  }
 } 

Note the FQDN

Create the KDC Database

# sudo kdb5_util create -s -r TEST.COM 

Confirm password twice Start and Enable Kerberos

# systemctl start krb5kdc kadmin 

Create Principals

# kadmin.local kadmin.local: addprinc root/admin 

quit

# sudo systemctl start krb5kdc kadmin 
# sudo systemctl enable krb5kdc kadmin 

On the Ambari UI enable Kerberos

root/admin@TEST.CH 

password {password_created_earlier}

This should take you through the procedure. Make sure the KDC is up and running !

View solution in original post

19 REPLIES 19

avatar
Master Mentor

@Sam Red

1) By the looks your /etc/krb5.conf doesn't look right. In the case your REALM is RELAY please follow strictly the below notice some are upper and lower case and also the .relay in teh domain_realm section.

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = RELAY.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  relay.com = RELAY.COM
  .relay.com = RELAY.COM
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  RELAY.COM = {
    admin_server = gulu.relay.com
    kdc = gulu.relay.com
  } 

2) I see many entries where do the come from its wrong, there shouldn't be any entries except if you have a kerberos HA setup for the failover KDC

kdc = IP 
kdc = IP 
kdc = IP 
kdc = IP 
kdc = IP 
kdc = IP 

3) Were the keytabs created correctly in /etc/security/keytabs/* ?

Can you grab a valid kerberos ticket in the below example I first list the principal available for user opera1 whos keytab is opera1.keytab

List principal for opera1

# klist -kt /etc/security/keytabs/opera1.keytab 
Keytab name: FILE:/etc/security/keytabs/opera1.keytab 
KVNO Timestamp Principal ---- ------------------- 
1 08/24/2017 18:28:26 opera1@RELAY.COM 

Grab a kerberos ticket

# kinit -kt /etc/security/keytabs/opera1.keytab opera1@RELAY.COM 

List the kerberos ticket to see validity

# klist 
Ticket cache: FILE:/tmp/krb5cc_0 
Default principal: opera1@RELAY.COM 
Valid starting          Expires           Service principal 
08/25/2017 20:33:43 08/26/2017 20:33:42 krbtgt/RELAY.COM@RELAY.COM 

Please let me know

avatar
Contributor

@Geoffrey Shelton Okot looks like there will be lot of mess up on my cluster. I follow same steps not able to do kinit.

kinit: Client 'host/*@RELAY.COM' not found in Kerberos database while getting initial credentials

How do i reset everything and re do from Scratch

avatar
Master Mentor

@Sam Red

Panic NOT 🙂
Just to be sure of the step you took.

1. Did the Ambari kerberization succeed with errors?

2. Can you paste the contents of /etc/security/keytabs/*

3. Can you list principals in the KDC database?

# kadmin.local
kadmin.local: listprincs

4.Did you make the changes I pointed out earlier

5. Can you share the files the below files remember to scramble the private details

/etc/krb5.conf
/var/kerberos/krb5kdc/kdc.conf
/var/kerberos/krb5kdc/kadm5.acl

6. Can you validate that service keytabs were create on all host in /etc/security/keytabs/* where service like YARN,Zookeeper or ranger are installed.

Please revert

avatar
Contributor

1. yes

2. -bash: cd: /etc/security/keytabs/*: No such file or directory

3. kadmin.local: Can not fetch master key (error: No such file or directory). while initializing kadmin.local interface

4. Yes

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
[realms]
 RELAY.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  


[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = RELAY.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5


[domain_realm]
  host.com = RELAY.COM
  .host.com = RELAY.COM 
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log


[realms]
  RELAY.COM = {
    admin_server = ambari and kerberos hostname where installed
    kdc = host
  }



* /admin@RELAY.COM *

avatar
Master Mentor

@Sam Red

Solution to issue No.2

If you kerberized the cluster using the Ambari tool then the keytabs MUST have been generated under /etc/security/keytabs can you validate that you have the keytabs by running the below command as root,you should see a couple of keytabs

# ls -al  /etc/security/keytabs

Solution to issue No.3

Notice you have to logon to kadmin interface as user root

If your previously run the sudo yum install -y krb5-server krb5-libs krb5-workstation then kadmin was installed see attached my intercation with kadmin my kdc host FQDN is osaka.test.com and my REALM is TEST

Solution to issue No.4

I can already see some error in your /etc/krb5.conf it should look like this. I have adjusted the correct entries but you need to change ONLY the 2 entries of FQDN_of_KDC with the below output from the server where you install the KDC server

# hostname -f

Your /etc/krb5.conf should look like this

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = RELAY.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  relay.com = RELAY.COM
  .relay.com = RELAY.COM 
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  RELAY.COM = {
    admin_server = FQDN_of_KDC
    kdc = FQDN_of_KDC
  }

Please let me know if you need !


samred.jpg

avatar
Contributor
@Geoffrey Shelton Okot

Thank you for your time. I tried with different ways. Now i am able to start kadmin and enabled too. Able to get generate Principals.

[root@Host krb5kdc]# kadmin.local
Authenticating as principal root/admin@RELAY.COM with password.
kadmin.local: 

when i do keytab file to generate

ls: cannot access /etc/security/keytabs: No such file or directory

I do Klist it says empty.

After I do all these I went to ambari and enable kerberos and selected as existing KDC and Principal as root/admin@RELAY.COM and password that we generated when we creating these principal.

Still in test client is is failed.

Error in KDC Host :

2017-08-29 11:04:22,427 - Failed to create principal, phddata-08291@RELAY.COM - Failed to create service principal for phddata-08291@RELAY.COM
STDOUT: Authenticating as principal root/admin@RELAY.COM with password.
Password for root/admin@RELAY.COM: 
Enter password for principal "phddata-08291@RELAY.COM": 
Re-enter password for principal "phddata-08297@RELAY.COM": 

STDERR: WARNING: no policy specified for phddata-08291@RELAY.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "phddata-08291@RELAY.COM".

avatar
Contributor

@Geoffrey Shelton Okot

Thank you. I am able to resolve the issue.

there is space between * / like these. I just edited it to */admin@RELAY.COM * then i am able to do test clients.

avatar
Master Mentor

@Sam Red

I am happy we have advanced at times it such trivial things like extra space or the [domain_realm] you forgot that messes us up but once we have gone through it, we have the memories 🙂

I am NOT yes sure why you can't find the keytabs , once again let's give it this try sequentially.

Once on the console as root check my steps

[root@gulu ~]# cd /etc/security/keytabs/
[root@gulu keytabs]# ls
activity-analyzer.headless.keytab  ams-hbase.regionserver.keytab  hbase.service.keytab    knox.service.keytab  rangeradmin.service.keytab     smokeuser.headless.keytab activity-explorer.headless.keytab  ams-zk.service.keytab

You should definitely see many keytabs here or try

# locate  hive.service.keytab

The issue of Failed to create principal, phddata-08291@RELAY.COM I tried adding the principal in my KDC see

# kadmin.local
Authenticating as principal root/admin@TEST.COM with password.
kadmin.local:  addprinc phddata-08291@TEST.COM
WARNING: no policy specified for phddata-08291@TEST.COM; defaulting to no policy
Enter password for principal "phddata-08291@TEST.COM": xxxxxx 

In the above, the password for phddata-08291@TEST.COM is the KDC password and you got to confirm it twice.

To generate the keytab for user phddata-08291 do the following as root an invoke the ktutil (keytab utility) on the KDC

# cd /etc/security/keytabs
# sudo ktutil 
ktutil: addent -password -p phddata-08291@TEST.COM -k 1 -e RC4-HMAC 
Password for phddata-08291@TEST.COM: xxxx
ktutil: wkt phddata-08291.keytab 
ktutil: q 
# chown phddata-08291:phddata-08291 phddata-08291.keytab

Infact you can copy and pass these commands in ktutil and give the password you earlier created, the phddata-08291.keytab

should be visible in /etc/security/keytabs but owned by root:root until you run the last chown command

I hope that helps you and remember to reward me if that works ,I am sure it will work

avatar
Contributor

@Geoffrey Shelton Okot

Thank You. I am able to resolve the issues and now we are finally kerberized the cluster.

avatar
Master Mentor

@Sam Red

I told you ....:-) now pat my back with a reward the beer is for next time !!!!! Enjoy Hadoop