question HDP Kerberos enable through Ambari in Support Questions

HDP Kerberos enable through Ambari

sunilreddykallu — Fri, 25 Aug 2017 05:47:52 GMT

Hi I have 6 node cluster setup as HDP 2.5 and ambari 2.4

Need to enable Kerberos Security:

Steps Did In ambari server :

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/_enabling_kerberos_security_in_ambari.html

2. Yum install Kerberos and clients in all nodes

3. check /etc/krb.conf and checked on all nodes same files and realms

4. checked acl.file and edited and saved and restarted Kdc again

5. Enabling kerberos in ambari it was throwing error in Test Clients:

Error message: An internal system exception occurred: The 'krb5-conf' configuration is not available

500 status codereceived on POST method for API: /api/v1/clusters/Cluster/requests

Re: HDP Kerberos enable through Ambari

Shelton — Fri, 25 Aug 2017 06:11:39 GMT

@Sam Red

Did you create an admin principal? Are the krb5.conf file in /etc/krb5.conf? I just did kerberize a HDP 2.6 an hour ago without any issue what is the OS?

We can quickly resolve that !

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Fri, 25 Aug 2017 06:15:38 GMT

@Geoffrey Shelton Okot Yes I did created principal as admin/admin@host.com and OS is centos 7

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Fri, 25 Aug 2017 06:21:50 GMT

@Geoffrey Shelton Okot If you can share step by step that much heplful to get done.

Re: HDP Kerberos enable through Ambari

Shelton — Fri, 25 Aug 2017 06:53:19 GMT

@Sam Red

On the KDC server which can be the same as the Ambari server or any server in the cluster install both rpm's on all

# yum install -y krb5-server krb5-workstation

Edit Server Configuration Files

cd  /var/kerberos/krb5kdc

Edit these 2 files

kdc.conf
kadm5.acl

Mine are save

# cat kadm5.acl 
*/admin@TEST.COM      *

# cat kdc.conf

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
[realms]
 TEST.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal

Edit the Client Configuration

This file should be available on the KDC server edit it and copy it to the same path to all the other nodes

# cat /etc/krb5.conf

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = TEST.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  test.com = TEST.COM
  .test.com = TEST.COM
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  TEST.COM = {
    admin_server = kdc.TEST.com
    kdc = kdc.test.com
  }
 }

Note the FQDN

Create the KDC Database

# sudo kdb5_util create -s -r TEST.COM

Confirm password twice Start and Enable Kerberos

# systemctl start krb5kdc kadmin

Create Principals

# kadmin.local kadmin.local: addprinc root/admin

quit

# sudo systemctl start krb5kdc kadmin

# sudo systemctl enable krb5kdc kadmin

On the Ambari UI enable Kerberos

root/admin@TEST.CH 

password {password_created_earlier}

This should take you through the procedure. Make sure the KDC is up and running !

Re: HDP Kerberos enable through Ambari

Shelton — Fri, 25 Aug 2017 06:54:51 GMT

@Sam Red

There you go !

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Fri, 25 Aug 2017 23:10:03 GMT

Thank You.

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Fri, 25 Aug 2017 23:58:18 GMT

@Geoffrey Shelton Okot followed all these steps when i am starting Krb5kdc and kadmin I am Getting error like these.

Job for krb5kdc.service failed because the control process exited with error code. See "systemctl status krb5kdc.service" and "journalctl -xe" for details.

Re: HDP Kerberos enable through Ambari

Shelton — Sat, 26 Aug 2017 00:38:12 GMT

@Sam Red

if you are root the remove the sudo !

What is the output of

# systemctl status krb5kdc.service

The result should be

# systemctl status krb5kdc.service
● krb5kdc.service - Kerberos 5 KDC
   Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-08-24 15:30:12 CEST; 1 day 4h ago
 Main PID: 23781 (krb5kdc)
   CGroup: /system.slice/krb5kdc.service
           └─23781 /usr/sbin/krb5kdc -P /var/run/krb5kdc.pid
Aug 24 15:30:12 bombay.test.com systemd[1]: Starting Kerberos 5 KDC...
Aug 24 15:30:12 bombay.test.com systemd[1]: Started Kerberos 5 KDC

Please let me know

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Sat, 26 Aug 2017 00:56:15 GMT

@Geoffrey Shelton Okot Thank you so much for your help. These are the configs i specified On ambari server

Configs here :

kdc.conf

[kdcdefaults]

kdc_ports = 88

kdc_tcp_ports = 88 [realms]

RELAY.COM = { #master_key_type = aes256-cts

acl_file = /var/kerberos/krb5kdc/kadm5.acl

dict_file = /usr/share/dict/words

admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab

cat kadm5.acl

* /admin@RELAY.COM *

[libdefaults]

renew_lifetime = 7d

forwardable = true default_realm = RELAY.COM

ticket_lifetime = 24h

dns_lookup_realm = false

dns_lookup_kdc = false

default_ccache_name = /tmp/krb5cc_%{uid}

#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5

[logging] default = FILE:/var/log/krb5kdc.log

admin_server = FILE:/var/log/kadmind.log

kdc = FILE:/var/log/krb5kdc.log

[realms]

RELAY.COM =

{ admin_server = RELAY.COM

kdc = IP

}

krb5kdc.service - Kerberos 5 KDC

Loaded: loaded (/usr/lib/systemd/system/krb5kdc.service; enabled; vendor preset: disabled)

Active: failed (Result: exit-code) since Fri 2017-08-25 10:53:48 PDT; 3s ago

Process: 22602 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5kdc.pid $KRB5KDC_ARGS (code=exited, status=1/FAILURE)

Main PID: 1911 (code=exited, status=0/SUCCESS)

Aug 25 10:53:48 systemd[1]: Starting Kerberos 5 KDC...

Aug 25 10:53:48 krb5kdc[22602]: krb5kdc: cannot initialize realm RELAY.COM - see log file for details

Aug 25 10:53:48 : krb5kdc.service: control process exited, code=exited status=1

Aug 25 10:53:48: Failed to start Kerberos 5 KDC.

Aug 25 10:53:48 Unit krb5kdc.service entered failed state.

Aug 25 10:53:48 krb5kdc.service failed.

Re: HDP Kerberos enable through Ambari

Shelton — Sat, 26 Aug 2017 01:47:46 GMT

@Sam Red

1) By the looks your /etc/krb5.conf doesn't look right. In the case your REALM is RELAY please follow strictly the below notice some are upper and lower case and also the .relay in teh domain_realm section.

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = RELAY.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  relay.com = RELAY.COM
  .relay.com = RELAY.COM
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  RELAY.COM = {
    admin_server = gulu.relay.com
    kdc = gulu.relay.com
  }

2) I see many entries where do the come from its wrong, there shouldn't be any entries except if you have a kerberos HA setup for the failover KDC

kdc = IP 
kdc = IP 
kdc = IP 
kdc = IP 
kdc = IP 
kdc = IP

3) Were the keytabs created correctly in /etc/security/keytabs/* ?

Can you grab a valid kerberos ticket in the below example I first list the principal available for user opera1 whos keytab is opera1.keytab

List principal for opera1

# klist -kt /etc/security/keytabs/opera1.keytab 
Keytab name: FILE:/etc/security/keytabs/opera1.keytab 
KVNO Timestamp Principal ---- ------------------- 
1 08/24/2017 18:28:26 opera1@RELAY.COM

Grab a kerberos ticket

# kinit -kt /etc/security/keytabs/opera1.keytab opera1@RELAY.COM

List the kerberos ticket to see validity

# klist 
Ticket cache: FILE:/tmp/krb5cc_0 
Default principal: opera1@RELAY.COM 
Valid starting          Expires           Service principal 
08/25/2017 20:33:43 08/26/2017 20:33:42 krbtgt/RELAY.COM@RELAY.COM

Please let me know

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Sat, 26 Aug 2017 03:55:01 GMT

@Geoffrey Shelton Okot looks like there will be lot of mess up on my cluster. I follow same steps not able to do kinit.

kinit: Client 'host/*@RELAY.COM' not found in Kerberos database while getting initial credentials

How do i reset everything and re do from Scratch

Re: HDP Kerberos enable through Ambari

Shelton — Sat, 26 Aug 2017 04:21:26 GMT

@Sam Red

Panic NOT 🙂
Just to be sure of the step you took.

1. Did the Ambari kerberization succeed with errors?

2. Can you paste the contents of /etc/security/keytabs/*

3. Can you list principals in the KDC database?

# kadmin.local
kadmin.local: listprincs

4.Did you make the changes I pointed out earlier

5. Can you share the files the below files remember to scramble the private details

/etc/krb5.conf
/var/kerberos/krb5kdc/kdc.conf
/var/kerberos/krb5kdc/kadm5.acl

6. Can you validate that service keytabs were create on all host in /etc/security/keytabs/* where service like YARN,Zookeeper or ranger are installed.

Please revert

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Tue, 29 Aug 2017 04:20:24 GMT

1. yes

2. -bash: cd: /etc/security/keytabs/*: No such file or directory

3. kadmin.local: Can not fetch master key (error: No such file or directory). while initializing kadmin.local interface

4. Yes

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
[realms]
 RELAY.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = RELAY.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5


[domain_realm]
  host.com = RELAY.COM
  .host.com = RELAY.COM 
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log


[realms]
  RELAY.COM = {
    admin_server = ambari and kerberos hostname where installed
    kdc = host
  }

* /admin@RELAY.COM *

Re: HDP Kerberos enable through Ambari

Shelton — Tue, 29 Aug 2017 14:33:39 GMT

@Sam Red

Solution to issue No.2

If you kerberized the cluster using the Ambari tool then the keytabs MUST have been generated under /etc/security/keytabs can you validate that you have the keytabs by running the below command as root,you should see a couple of keytabs

# ls -al  /etc/security/keytabs

Solution to issue No.3

Notice you have to logon to kadmin interface as user root

If your previously run the sudo yum install -y krb5-server krb5-libs krb5-workstation then kadmin was installed see attached my intercation with kadmin my kdc host FQDN is osaka.test.com and my REALM is TEST

Solution to issue No.4

I can already see some error in your /etc/krb5.conf it should look like this. I have adjusted the correct entries but you need to change ONLY the 2 entries of FQDN_of_KDC with the below output from the server where you install the KDC server

# hostname -f

Your /etc/krb5.conf should look like this

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = RELAY.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  relay.com = RELAY.COM
  .relay.com = RELAY.COM 
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  RELAY.COM = {
    admin_server = FQDN_of_KDC
    kdc = FQDN_of_KDC
  }

Please let me know if you need !

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Wed, 30 Aug 2017 01:08:11 GMT

@Geoffrey Shelton Okot

Thank you for your time. I tried with different ways. Now i am able to start kadmin and enabled too. Able to get generate Principals.

[root@Host krb5kdc]# kadmin.local
Authenticating as principal root/admin@RELAY.COM with password.
kadmin.local:

when i do keytab file to generate

ls: cannot access /etc/security/keytabs: No such file or directory

I do Klist it says empty.

After I do all these I went to ambari and enable kerberos and selected as existing KDC and Principal as root/admin@RELAY.COM and password that we generated when we creating these principal.

Still in test client is is failed.

Error in KDC Host :

2017-08-29 11:04:22,427 - Failed to create principal, phddata-08291@RELAY.COM - Failed to create service principal for phddata-08291@RELAY.COM
STDOUT: Authenticating as principal root/admin@RELAY.COM with password.
Password for root/admin@RELAY.COM: 
Enter password for principal "phddata-08291@RELAY.COM": 
Re-enter password for principal "phddata-08297@RELAY.COM": 

STDERR: WARNING: no policy specified for phddata-08291@RELAY.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "phddata-08291@RELAY.COM".

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Wed, 30 Aug 2017 01:48:26 GMT

@Geoffrey Shelton Okot

Thank you. I am able to resolve the issue.

there is space between * / like these. I just edited it to */admin@RELAY.COM * then i am able to do test clients.

Re: HDP Kerberos enable through Ambari

Shelton — Wed, 30 Aug 2017 02:52:11 GMT

@Sam Red

I am happy we have advanced at times it such trivial things like extra space or the [domain_realm] you forgot that messes us up but once we have gone through it, we have the memories 🙂

I am NOT yes sure why you can't find the keytabs , once again let's give it this try sequentially.

Once on the console as root check my steps

[root@gulu ~]# cd /etc/security/keytabs/
[root@gulu keytabs]# ls
activity-analyzer.headless.keytab  ams-hbase.regionserver.keytab  hbase.service.keytab    knox.service.keytab  rangeradmin.service.keytab     smokeuser.headless.keytab activity-explorer.headless.keytab  ams-zk.service.keytab

You should definitely see many keytabs here or try

# locate  hive.service.keytab

The issue of Failed to create principal, phddata-08291@RELAY.COM I tried adding the principal in my KDC see

# kadmin.local
Authenticating as principal root/admin@TEST.COM with password.
kadmin.local:  addprinc phddata-08291@TEST.COM
WARNING: no policy specified for phddata-08291@TEST.COM; defaulting to no policy
Enter password for principal "phddata-08291@TEST.COM": xxxxxx

In the above, the password for phddata-08291@TEST.COM is the KDC password and you got to confirm it twice.

To generate the keytab for user phddata-08291 do the following as root an invoke the ktutil (keytab utility) on the KDC

# cd /etc/security/keytabs
# sudo ktutil 
ktutil: addent -password -p phddata-08291@TEST.COM -k 1 -e RC4-HMAC 
Password for phddata-08291@TEST.COM: xxxx
ktutil: wkt phddata-08291.keytab 
ktutil: q 
# chown phddata-08291:phddata-08291 phddata-08291.keytab

Infact you can copy and pass these commands in ktutil and give the password you earlier created, the phddata-08291.keytab

should be visible in /etc/security/keytabs but owned by root:root until you run the last chown command

I hope that helps you and remember to reward me if that works ,I am sure it will work

Re: HDP Kerberos enable through Ambari

sunilreddykallu — Wed, 30 Aug 2017 03:07:35 GMT

@Geoffrey Shelton Okot

Thank You. I am able to resolve the issues and now we are finally kerberized the cluster.

Re: HDP Kerberos enable through Ambari

Shelton — Wed, 30 Aug 2017 03:10:03 GMT

@Sam Red

I told you ....:-) now pat my back with a reward the beer is for next time !!!!! Enjoy Hadoop