Created 08-24-2017 10:47 PM
Hi I have 6 node cluster setup as HDP 2.5 and ambari 2.4
Need to enable Kerberos Security:
Steps Did In ambari server :
1.
2. Yum install Kerberos and clients in all nodes
3. check /etc/krb.conf and checked on all nodes same files and realms
4. checked acl.file and edited and saved and restarted Kdc again
5. Enabling kerberos in ambari it was throwing error in Test Clients:
Error message: An internal system exception occurred: The 'krb5-conf' configuration is not available
500 status codereceived on POST method for API: /api/v1/clusters/Cluster/requests
Created 08-24-2017 11:53 PM
@Sam Red
On the KDC server which can be the same as the Ambari server or any server in the cluster install both rpm's on all
# yum install -y krb5-server krb5-workstation
Edit Server Configuration Files
cd /var/kerberos/krb5kdc
Edit these 2 files
kdc.conf kadm5.acl
Mine are save
# cat kadm5.acl */admin@TEST.COM *
# cat kdc.conf
[kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] TEST.COM = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
Edit the Client Configuration
This file should be available on the KDC server edit it and copy it to the same path to all the other nodes
# cat /etc/krb5.conf
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = TEST.COM ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [domain_realm] test.com = TEST.COM .test.com = TEST.COM [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] TEST.COM = { admin_server = kdc.TEST.com kdc = kdc.test.com } }
Note the FQDN
Create the KDC Database
# sudo kdb5_util create -s -r TEST.COM
Confirm password twice Start and Enable Kerberos
# systemctl start krb5kdc kadmin
Create Principals
# kadmin.local kadmin.local: addprinc root/admin
quit
# sudo systemctl start krb5kdc kadmin
# sudo systemctl enable krb5kdc kadmin
On the Ambari UI enable Kerberos
root/admin@TEST.CH password {password_created_earlier}
This should take you through the procedure. Make sure the KDC is up and running !
Created 08-25-2017 06:47 PM
1) By the looks your /etc/krb5.conf doesn't look right. In the case your REALM is RELAY please follow strictly the below notice some are upper and lower case and also the .relay in teh domain_realm section.
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = RELAY.COM ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [domain_realm] relay.com = RELAY.COM .relay.com = RELAY.COM [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] RELAY.COM = { admin_server = gulu.relay.com kdc = gulu.relay.com }
2) I see many entries where do the come from its wrong, there shouldn't be any entries except if you have a kerberos HA setup for the failover KDC
kdc = IP kdc = IP kdc = IP kdc = IP kdc = IP kdc = IP
3) Were the keytabs created correctly in /etc/security/keytabs/* ?
Can you grab a valid kerberos ticket in the below example I first list the principal available for user opera1 whos keytab is opera1.keytab
List principal for opera1
# klist -kt /etc/security/keytabs/opera1.keytab Keytab name: FILE:/etc/security/keytabs/opera1.keytab KVNO Timestamp Principal ---- ------------------- 1 08/24/2017 18:28:26 opera1@RELAY.COM
Grab a kerberos ticket
# kinit -kt /etc/security/keytabs/opera1.keytab opera1@RELAY.COM
List the kerberos ticket to see validity
# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: opera1@RELAY.COM Valid starting Expires Service principal 08/25/2017 20:33:43 08/26/2017 20:33:42 krbtgt/RELAY.COM@RELAY.COM
Please let me know
Created 08-25-2017 08:55 PM
@Geoffrey Shelton Okot looks like there will be lot of mess up on my cluster. I follow same steps not able to do kinit.
kinit: Client 'host/*@RELAY.COM' not found in Kerberos database while getting initial credentials
How do i reset everything and re do from Scratch
Created 08-25-2017 09:21 PM
Panic NOT 🙂
Just to be sure of the step you took.
1. Did the Ambari kerberization succeed with errors?
2. Can you paste the contents of /etc/security/keytabs/*
3. Can you list principals in the KDC database?
# kadmin.local kadmin.local: listprincs
4.Did you make the changes I pointed out earlier
5. Can you share the files the below files remember to scramble the private details
/etc/krb5.conf /var/kerberos/krb5kdc/kdc.conf /var/kerberos/krb5kdc/kadm5.acl
6. Can you validate that service keytabs were create on all host in /etc/security/keytabs/* where service like YARN,Zookeeper or ranger are installed.
Please revert
Created 08-28-2017 09:20 PM
1. yes
2. -bash: cd: /etc/security/keytabs/*: No such file or directory
3. kadmin.local: Can not fetch master key (error: No such file or directory). while initializing kadmin.local interface
4. Yes
[kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] RELAY.COM = { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = RELAY.COM ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [domain_realm] host.com = RELAY.COM .host.com = RELAY.COM [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] RELAY.COM = { admin_server = ambari and kerberos hostname where installed kdc = host }
* /admin@RELAY.COM *
Created 08-29-2017 07:33 AM
Solution to issue No.2
If you kerberized the cluster using the Ambari tool then the keytabs MUST have been generated under /etc/security/keytabs can you validate that you have the keytabs by running the below command as root,you should see a couple of keytabs
# ls -al /etc/security/keytabs
Solution to issue No.3
Notice you have to logon to kadmin interface as user root
If your previously run the sudo yum install -y krb5-server krb5-libs krb5-workstation then kadmin was installed see attached my intercation with kadmin my kdc host FQDN is osaka.test.com and my REALM is TEST
Solution to issue No.4
I can already see some error in your /etc/krb5.conf it should look like this. I have adjusted the correct entries but you need to change ONLY the 2 entries of FQDN_of_KDC with the below output from the server where you install the KDC server
# hostname -f
Your /etc/krb5.conf should look like this
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = RELAY.COM ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [domain_realm] relay.com = RELAY.COM .relay.com = RELAY.COM [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] RELAY.COM = { admin_server = FQDN_of_KDC kdc = FQDN_of_KDC }
Please let me know if you need !
Created 08-29-2017 06:08 PM
Thank you for your time. I tried with different ways. Now i am able to start kadmin and enabled too. Able to get generate Principals.
[root@Host krb5kdc]# kadmin.local Authenticating as principal root/admin@RELAY.COM with password. kadmin.local:
when i do keytab file to generate
ls: cannot access /etc/security/keytabs: No such file or directory
I do Klist it says empty.
After I do all these I went to ambari and enable kerberos and selected as existing KDC and Principal as root/admin@RELAY.COM and password that we generated when we creating these principal.
Still in test client is is failed.
Error in KDC Host :
2017-08-29 11:04:22,427 - Failed to create principal, phddata-08291@RELAY.COM - Failed to create service principal for phddata-08291@RELAY.COM STDOUT: Authenticating as principal root/admin@RELAY.COM with password. Password for root/admin@RELAY.COM: Enter password for principal "phddata-08291@RELAY.COM": Re-enter password for principal "phddata-08297@RELAY.COM": STDERR: WARNING: no policy specified for phddata-08291@RELAY.COM; defaulting to no policy add_principal: Operation requires ``add'' privilege while creating "phddata-08291@RELAY.COM".
Created 08-29-2017 06:48 PM
Thank you. I am able to resolve the issue.
there is space between * / like these. I just edited it to */admin@RELAY.COM * then i am able to do test clients.
Created 08-29-2017 07:52 PM
I am happy we have advanced at times it such trivial things like extra space or the [domain_realm] you forgot that messes us up but once we have gone through it, we have the memories 🙂
I am NOT yes sure why you can't find the keytabs , once again let's give it this try sequentially.
Once on the console as root check my steps
[root@gulu ~]# cd /etc/security/keytabs/ [root@gulu keytabs]# ls activity-analyzer.headless.keytab ams-hbase.regionserver.keytab hbase.service.keytab knox.service.keytab rangeradmin.service.keytab smokeuser.headless.keytab activity-explorer.headless.keytab ams-zk.service.keytab
You should definitely see many keytabs here or try
# locate hive.service.keytab
The issue of Failed to create principal, phddata-08291@RELAY.COM I tried adding the principal in my KDC see
# kadmin.local Authenticating as principal root/admin@TEST.COM with password. kadmin.local: addprinc phddata-08291@TEST.COM WARNING: no policy specified for phddata-08291@TEST.COM; defaulting to no policy Enter password for principal "phddata-08291@TEST.COM": xxxxxx
In the above, the password for phddata-08291@TEST.COM is the KDC password and you got to confirm it twice.
To generate the keytab for user phddata-08291 do the following as root an invoke the ktutil (keytab utility) on the KDC
# cd /etc/security/keytabs # sudo ktutil ktutil: addent -password -p phddata-08291@TEST.COM -k 1 -e RC4-HMAC Password for phddata-08291@TEST.COM: xxxx ktutil: wkt phddata-08291.keytab ktutil: q # chown phddata-08291:phddata-08291 phddata-08291.keytab
Infact you can copy and pass these commands in ktutil and give the password you earlier created, the phddata-08291.keytab
should be visible in /etc/security/keytabs but owned by root:root until you run the last chown command
I hope that helps you and remember to reward me if that works ,I am sure it will work
Created 08-29-2017 08:07 PM
Thank You. I am able to resolve the issues and now we are finally kerberized the cluster.
Created 08-29-2017 08:10 PM
I told you ....:-) now pat my back with a reward the beer is for next time !!!!! Enjoy Hadoop