Connecting to HBase in a Kerberos Enabled Cluster

[#1]

This article will show a step by step guide on how to connect to HBase in an a Kerberos enable cluster. If you need to setup a cluster that have Kerberos enabled, this is a good guid to follow.

Clone Example Project

This article will be based on this project. Please run the following to clone the project.

$ git clone https://github.com/jjmeyer0/hdp-test-examples

Creating Keytab

Before getting into the code, it is important to generate necessary files. If a key tab is not available follow the steps below to create one. In the example below, a key tab for the user jj and realm EXAMPLE.COM is created. The below commands should be run on one of the nodes in the cluster.

$ kadmin.local
$ addprinc jj@EXAMPLE.COM
$ <CTRL-D>
$ ktutil
$ addent -password -p jj -k 1 -e RC4-HMAC
$ wkt jj.keytab
$ q

Preparing User in HBase

The user that was used above must be given correct permissions in HBase. To do so do the following:

$ hbase shell
hbase(main):001:0> grant 'jj', 'RW'

Obtaining Necessary Files

This example also expects the files listed below. Below is a walkthrough on how to copy the necessary files from the cluster to local.

• hbase-site.xml
• <username>.keytab
• krb5.conf

$ scp -i <insecure_private_key> vagrant@c6401:/etc/krb5.conf .
$ scp -i <insecure_private_key> vagrant@c6401:/etc/hbase/conf/hbase-site.xml .
$ scp -i <insecure_private_key> root@c6401:~/jj.keytab .

Once the files have been obtained, please move them to the following directory.

src/main/resources/

For testing, it is recommended to change 'hbase.client.retries.number' property in hbase-site.xml. By default it is 35. This is quite high when running some tests.

Code Walkthrough

The First thing that needs to be done is to create and load the HBase configuration.

// Setting up the HBase configuration
Configuration configuration = new Configuration();
configuration.addResource("src/main/resources/hbase-site.xml");

Next point to the krb5.conf file and setup the kerberos principal and keytab.

// Point to the krb5.conf file.
System.setProperty("java.security.krb5.conf", "src/main/resources/krb5.conf");
System.setProperty("sun.security.krb5.debug", "true");

// Override these values by setting -DkerberosPrincipal and/or -DkerberosKeytab
String principal = System.getProperty("kerberosPrincipal", "jj@EXAMPLE.COM");
String keytabLocation = System.getProperty("kerberosKeytab", "src/main/resources/jj.keytab");

Now login with the principal and keytab defined above.

UserGroupInformation.setConfiguration(configuration);
UserGroupInformation.loginUserFromKeytab(principal, keytabLocation)

Please see this file for full example. A Scala version can be found here.

Resources

1. Ambari Quickstart Guide
2. Full Code (This article covers HBase examples)

[#2]

@JJ Meyer

I've a HBase client application which exactly does the same as your code but for some unknown reason Kerberos authentication fails with this exception "GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]". I get this exception only when Kerberos ticket cache is empty but if I do a "Kinit" for my principal before starting the application it runs fine. This indicates Krb5LoginModule's option useTicketCache is set to true somewhere but using "UserGroupInformation.loginUserFromKeytab(principal, keytabLocation)" internally creates a JAAS config which would have useTicketCache as false. What would be the possible reason for this behaviour? From where does my client is picking up useTicketCache as true? I do have a JAAS config file with useTicketCache as true in HBASE_HOME/conf but I'm not using this anywhere in my client application, would my HBase client dependencies pick this configuration? Please let me know what could be the possible reason for this. Regards, Niranjan

[#3]

FYR: https://bryanbende.com/development/2016/03/20/learning-kerberos-and-ugi

[#4]

https://stackoverflow.com/questions/31233830/apache-spark-setting-spark-eventlog-enabled-and-spark-e...

http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-job-fails-in-cluster-mode/td-...

if you want to use the keytab from inside a spark (cluster mode), copy the files using --files option and ALIAS. Then use relative path ./your.keytab