Kerberos Enable first time

[#1]

Hi,

I have HDP installed on my cluster.Now next task is to enable kerberos for HDP cluster .Can some one point me to step by step documentation please?Do i need AD/LDAP as well.Its a development cluster .

[#2]

@Mudit Kumar, You can refer this document : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/configuring_amb_hdp_for...

[#3]

@Mudit Kumar

Below is an outline of the next procedure

Assumption:

• - Centos6 or RHEL 6
  
• - REALM is EXAMPLE.COM
  

the command will differ for Centos/RHEL7 ie systemctl

# Install a new MIT KDC

Install a new version of the KDC server:

# yum install krb5-server krb5-libs krb5-workstation 

On KDC clients cluster clients datanodes etc

# yum install krb5-workstation 

# Edit the KDC server configuration file

Change the [realms] section of this file by replacing the default “kerberos.example.com” setting for the kdc and admin_server properties with the Fully Qualified Domain Name of the KDC server host. In the following example, “kerberos.example.com” has been replaced with “my.kdc.server”.

# vi /etc/krb5.conf

[realms]
 EXAMPLE.COM = {
   kdc = my.kdc.server
   admin_server = my.kdc.server
} 

Some components such as long-running spark jobs require renewable tickets. To configure MIT KDC to support them, ensure the following settings are specified in the libdefaults section of the /etc/krb5.conf file.

renew_lifetime = 7d 

# Create the Kerberos Database takes a while

 # kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM',
master key name 'K/M@EXAMPLE.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:xxxxxxxx {dont lose this password}
Re-enter KDC database master key to verify:xxxxxxxx 

# Start the KDC

Start the KDC server and the KDC admin server.

# service krb5kdc start
# service kadmin start 

# Set up the KDC server to auto-start on boot.

# chkconfig krb5kdc on
# chkconfig kadmin on 

# Create a Kerberos Admin

Create a KDC admin by creating an admin principal.

# kadmin.local -q "addprinc admin/admin" 
Authenticating as principal admin/admin@EXAMPLE.COM with password. 
WARNING: no policy specified for admin/admin@EXAMPLE.COM; defaulting to no policy 
Enter password for principal "admin/admin@EXAMPLE.COM": 
Re-enter password for principal "admin/admin@EXAMPLE.COM": Principal "admin/admin@EXAMPLE.COM" created. 

Confirm that this admin principal has permissions in the KDC ACL. Using a text editor,

open the KDC ACL file: /var/kerberos/krb5kdc/kadm5.acl Ensure that the KDC ACL file includes an entry so to allow the admin principal to administer the KDC for your specific realm. When using a realm that is different than EXAMPLE.COM, be sure there is an entry for the realm you are using. If not present, principal creation will fail. For example, for an admin/admin@HADOOP.COM principal, you should have an entry:

*/admin@EXAMPLE.COM * 

After editing and saving the kadm5.acl file, you must restart the kadmin process. RHEL/CentOS/Oracle Linux 6

# service kadmin restart 

Check status

# service krb5kdc status 

desired output krb5kdc (pid 2204) is running...

# service kadmin status 

desired output kadmind (pid 16891) is running...

# Install the JCE

On the Ambari Server, obtain the JCE policy file appropriate for the JDK version in your cluster. For Oracle JDK 1.8:

nstall JCE 8wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jce/8/jce_policy-8.zip"unzip jce_policy-8.zip

Save the policy file archive in a temporary location.On Ambari Server and on each host in the cluster, add the unlimited security policy JCE jars to $JAVA_HOME/jre/lib/security/.

# unzip -o -j -q jce_policy-8.zip -d /usr/jdk64/jdk1.8.0_77/jre/lib/security/ 

# Restart Ambari Server.

# ambari-server restart 

# Running the Kerberos Security Wizard

When choosing Existing MIT KDC or Existing Active Directory, the Kerberos Wizard prompts for information related to the KDC, the KDC Admin Account and the Service and Ambari principals. Once provided, Ambari will automatically create principals, generate keytabs and distribute keytabs to the hosts in the cluster. The services will be configured for Kerberos and the service components are restarted to authenticate against the KDC

# To continue

http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-security/content/ch_advanced_secu...

Go to Ambari GUI

To enable kerberos, the inputs are quite straight forward Admin pricipal password anREALM etc

Good luck

After the successful installation, all the service are restart !

Test without kerberos ticket ad HDFS user

su - hdfs 

Destroy any valid ticket

$kdestroy

The below command should error out

hdfs dfs -ls /user

List the generated keytabs

$ ls /etc/security/keytabs 

Test the with a valid Kerberos ticket as hdfs

$ klist -kt /etc/security/keytabs/hdfs.service.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 02/02/17 23:00:12 hdfs/london.EXAMPLE.COM@EXAMPLE.COM
   1 02/02/17 23:00:12 hdfs/london.EXAMPLE.COM@EXAMPLE.COM
   1 02/02/17 23:00:12 hdfs/london.EXAMPLE.COM@EXAMPLE.COM
   1 02/02/17 23:00:12 hdfs/london.EXAMPLE.COM@EXAMPLE.COM
   1 02/02/17 23:00:12 hdfs/london.EXAMPLE.COM@EXAMPLE.COM 

Get a ticket

$ kinit -kt /etc/security/keytabs/hdfs.service.keytab hdfs/london.EXAMPLE.COM@EXAMPLE.COM 

You should see a valid ticket

$ klist 
Ticket cache: FILE:/tmp/krb5cc_504 
Default principal: hdfs/london.EXAMPLE.COM@EXAMPLE.COM 

Valid starting           Expires           Service principal 
02/10/17 01:32:45        02/11/17 01:32:45 krbtgt/EXAMPLE.COM@EXAMPLE.COM 
renew until 02/10/17 01:32:45

The below command should succeed

hdfs dfs -ls /user

Hope that helps

[#4]

@Geoffrey Shelton Okot

Thanks!So we need to have MIT KDC or AD running.Right?

For the development environment,is there a way to setup MIT KDC specifically for development environment?Any link for that please?

[#5]

@Geoffrey Shelton Okot @Sandeep Nemuri I donot see kerberos wizard on my ambari?whats the issue?I have reached till the steps provided by @Geoffrey Shelton Okot till installiing JCE files and restrarting ambari server after that.

[#6]

i am able to find kerberos wizard ,sorry for the trouble.

[#7]

@Geoffrey Shelton @Sandeep Nemuri GUys,thanks a lot.I am done successfully.Can you share few steps to verify steps for services like hdfs,spark,yarn,hive,hbase!

[#8]

@Mudit Kumar

Testing is straightforward

Without Kerberos ticket

From ROOT switch to user hdfs

 # su - hdfs 

Check if hdfs has a ticket

$ klist
Ticket cache: FILE:/tmp/krb5cc_507
Default principal: hdfs-London@{YOUR_REALM}
Valid starting     Expires            Service principal
xx/xx/xx xx:xx:xx  xx/xx/xx xx:xx:xx  krbtgt/{YOUR_REALM}@{YOUR_REALM} 		renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx  xx/xx/xx xx:xx:xx  HTTP/{host_name}@{YOUR_REALM}		renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx  xx/xx/xx xx:xx:xx  HTTP/{host_name}@{YOUR_REALM} 	        renew until xx/xx/xx xx:xx:xx

If you see some out like above that means hdfs user was already given a ticket, destroy the ticket

$ kdestroy 

Now try accessing hdfs directory /user home

$ hdfs dfs -ls /user 

This should throughout an error

Test with Kerberos

To get a valid Kerberos ticket need to know the principal, it's the part that starts with hdfs-{xxx}

$ klist -ket /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (des3-cbc-sha1)
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (aes058-cts-hmac-sha1-96)
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (arcfour-hmac)
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (des-cbc-md5)
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (aes256-cts-hmac-sha1-96) 

Note the difference with below command it won't give you the encryption

 $ klist -kt /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} 

Grab a ticket, note I switch to kinit NOT klist and I append the principal hdfs-{host_name}@{YOUR_REALM} to the keytab

$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-{host_name}@{YOUR_REALM} 

Now I should have a valid ticket as shown below

$ klist
Ticket cache: FILE:/tmp/krb5cc_507
Default principal: hdfs-London@{YOUR_REALM}
Valid starting     Expires            Service principal
xx/xx/xx xx:xx:xx  xx/xx/xx xx:xx:xx  krbtgt/{YOUR_REALM}@{YOUR_REALM} 		renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx  xx/xx/xx xx:xx:xx  HTTP/{host_name}@{YOUR_REALM}		renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx  xx/xx/xx xx:xx:xx  HTTP/{host_name}@{YOUR_REALM} 	        renew until xx/xx/xx xx:xx:xx

Now I should be able to list the hdfs /user directory see the example below

$ hdfs dfs -ls / 
Found 11 items 
drwxrwxrwx - yarn hadoop 0 2018-05-09 21:45 /app-logs 
.......
.......
drwxrwxrwx - mapred hadoop 0 2018-05-14 14:19 /mr-history

Voila you are done,

So no service /user without a valid ticket can run any job on your cluster.

So can you Accept the answer I gave by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for these kinds of errors.

[#9]

Thanks @Geoffrey Shelton Okot . This is done.

How can i validate hive and hbase as well?

[#10 · Accepted solution]

@Mudit Kumar

Testing for HIVE or HBASE straightforward too just similar to the previous

Without Kerberos ticket

From ROOT switch to user hive/hbase

# su - hive 

or

# su - hbase 

Check if hdfs has a ticket

$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_1013) 

If you see some output different from the above destroy the ticket

$ kdestroy 

Try accessing hive or Hbase shell CLI

$ hive 

or

$hbase-shell 

When you press "ENTER" this should give you an error for both hive and hbase

Test with kerberos for hive

$ klist -ket /etc/security/keytabs/hive.keytab
Keytab name: FILE:/etc/security/keytabs/hive.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (des3-cbc-sha1)
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (aes058-cts-hmac-sha1-96)
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (arcfour-hmac)
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (des-cbc-md5)
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (aes256-cts-hmac-sha1-96) 

Test with kerberos for hbase

$ klist -ket /etc/security/keytabs/hbase.keytab
Keytab name: FILE:/etc/security/keytabs/hbase.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (des3-cbc-sha1)
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (aes058-cts-hmac-sha1-96)
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (arcfour-hmac)
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (des-cbc-md5)
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (aes256-cts-hmac-sha1-96) 

Note the difference with below command it won't give you the encryption

Check principal for hive

$ klist -kt /etc/security/keytabs/hive.keytab
Keytab name: FILE:/etc/security/keytabs/hive.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} 

Check principal for hbase

 $ klist -kt /etc/security/keytabs/hbase.keytab
Keytab name: FILE:/etc/security/keytabs/hbase.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} 
   1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} 

Grab a ticket, note I switch to kinit NOT klist and I append the principal for

hive-{host_name}@{YOUR_REALM} or hbase-{host_name}@{YOUR_REALM} to the keytab

$kinit -kt /etc/security/keytabs/hive.keytab hive-{host_name}@{YOUR_REALM} 

or

$kinit -kt /etc/security/keytabs/hbase.keytab hbase-{host_name}@{YOUR_REALM} 

Now I should have a valid ticket as shown below for either hive or hbase

$ klist 
Ticket cache: FILE:/tmp/krb5cc_507 
Default principal: hive-{host_name}@{YOUR_REALM} 
Valid starting 		Expires 	   Service principal 
xx/xx/xx xx:xx:xx 	xx/xx/xx xx:xx:xx  krbtgt/{YOUR_REALM}@{YOUR_REALM}   renew until xx/xx/xx xx:xx:xx 

Now I should be able to connect to hive CLI

$ hive 

After some time you should have the below output

$ hive
WARNING: Use "yarn jar" to launch YARN applications
................
Logging initialized using configuration in file:/etc/hive/2.5.0.0-817/0/hive-log4j.properties
hive>

Now you can run all the hive queries

For hbase

$ klist 
Ticket cache: FILE:/tmp/krb5cc_507 
Default principal: hbase-{host_name}@{YOUR_REALM} 
Valid starting 		Expires Service 	principal 
xx/xx/xx xx:xx:xx 	xx/xx/xx xx:xx:xx 	krbtgt/{YOUR_REALM}@{YOUR_REALM}renew until xx/xx/xx xx:xx:xx 

Now I should be able to connect to hbase shell and there shouldn't be any error

$ hbase shell 
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.94.23, rf42302b28aceaab773b15f234aa8718fff7eea3c, Thursday  May 10 18:54:09 UTC 2018
hbase(main):001:0>

Voila, you are done.

As reiterated please "Accept " and close the thread. You can open a new thread as this has become long 🙂