Oozie HA configuration with Kerberos

[#1]

Please follow below steps to setup Oozie HA configuration with Kerberos environment.

.

Step 1: Configure mysql/oracle database for Oozie as HA configuration does not work with default embedded Derby Database.

.

Please refer https://community.hortonworks.com/articles/183/moving-oozie-to-mysql-with-ambari.html for steps to migrate Oozie database.

.

Step 2: Login to Ambari UI, goto hosts, select host on which you need to add additional Oozie server, Click on Add and select Oozie server.

.

Please refer below screenshot, for e.g. I will add oozie server on kk3.hwxblr.com

.

Step 3: Setup Load balancer

Please refer this blogpost for setting up lightweight open source linux based load balancer.

.

Step4: Configure Kerberos for your cluster if not already done.

Please refer our blog for automated Kerberos configuration.

.

Step 5: Login to Ambari UI and set below configuration parameters for Oozie service.

oozie.zookeeper.connection.string=<zookeeper1>:2181,<zookeeper2>:2181,<zookeeper3>:2181

oozie.services.ext=org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService

oozie.base.url=http://<loadbalancer.hostname>:11000/oozie

oozie.authentication.kerberos.principal=*

.

Step 6: In oozie-env section of Oozie configuration, uncomment OOZIE_BASE_URL property and set it to http://<load-balancer-host>:11000/oozie

for example:

export OOZIE_BASE_URL="http://<loadbalance.hostname>:11000/oozie"

.

Step 7: Login to your KDC and create HTTP principal for load balancer.

kadmin.local -q "addprinc -randkey HTTP/<loadbalancer_hostname>@<realm>"

.

Step 8: Create a single spnego.service.keytab with both Oozie server's + Load balancer's principal and distribute the same on both the Oozie servers.

For example: In my case I have test1-ambari-server.hwxblr.com as loadbalancer and kk2/kk4 are my oozie servers

[root@kk4 ~]# klist -ket /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
 3 05/03/16 16:42:43 HTTP/kk4.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96)
 3 05/03/16 16:42:43 HTTP/kk4.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96)
 3 05/03/16 16:42:43 HTTP/kk4.hwxblr.com@HWX.COM (des3-cbc-sha1)
 3 05/03/16 16:42:43 HTTP/kk4.hwxblr.com@HWX.COM (arcfour-hmac)
 3 05/03/16 16:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96)
 3 05/03/16 16:44:05 HTTP/kk2.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96)
 3 05/03/16 16:44:05 HTTP/kk2.hwxblr.com@HWX.COM (des3-cbc-sha1)
 3 05/03/16 16:44:05 HTTP/kk2.hwxblr.com@HWX.COM (arcfour-hmac)
 4 05/03/16 16:43:18 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes256-cts-hmac-sha1-96)
 4 05/03/16 16:43:18 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (aes128-cts-hmac-sha1-96)
 4 05/03/16 16:43:18 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (des3-cbc-sha1)
 4 05/03/16 16:43:18 HTTP/test1-ambari-server.hwxblr.com@HWX.COM (arcfour-hmac)

.

Step 9: Make sure you have saved updated keytab on both the Oozie hosts.

.

Step 10: Restart Oozie services via Ambari UI

.

Step 11: Configure your browser for spnego authentication using steps given at below URLs

http://www.ghostar.org/2015/06/google-chrome-spnego-and-webhdfs-on-hadoop/

http://www.microhowto.info/howto/configure_firefox_to_authenticate_using_spnego_and_kerberos.html

.

Step 12: Hit http://<load-balancer-hostname>:11000/oozie and you should be able to see oozie UI

.

.

Please comment if you have any feedback/questions/suggestions. Happy Hadooping!!

[#2]

hi :

I followed your steps to setup OOZIE HA with kerberos environment

But my ambari GUI will have two alerts about Oozie Server Status, l

<pre>

Execution of 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie http://oozie-server1:11000/oozie -status' returned 255. Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, status: 403, message: Forbidden

</pre>

I use the command 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie http://oozie-server1:11000/oozie -status' to run on the physical node, it failed.

but I change the oozie server to my load balancer hostname, 'source /usr/hdp/current/oozie-server/conf/oozie-env.sh ; oozie admin -oozie http://loadbalancer.net:11000/oozie -status'

It will display result : 'System mode: NORMAL'

I think this is right.

do you meet the question? why ambari do not catch my load balancer hostname, it is still use original oozie server node to check the service?

thanks.

[#3]

Hi,

Thanks for this guide

i followed exacty how you explained but i still getting this error

Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Authentication failed, URL: http://loadbalancer_FQDN:11000/oozie/versions?user.name=oozie, status: 403, message: Forbidden

Any idea

Thanks