This is a guide to walk a user through enabling high availability for Oozie in an Ambari-managed cluster in which Kerberos has been enabled. It is assume that Ambari 2.4.0 (or above) is installed and Kerberos has been previously set up. However, if Kerberos has not yet been set up, the Kerberos-related items can be performed when Kerberos is enabled. Also, it is expected that an Oozie HA supported database is configured. A database such as MySQL or Oracle needs to have been configured since the Derby database is not sufficient.
Motivation
Most documentation properly shows the steps to enable Oozie HA; however when Kerberos is involved, some configuration properties need to be set in an alternate way. According some documentation, when HA is setup in an environment where Kerberos is enabled, the following properties must be set:
oozie-site/oozie.authentication.kerberos.principal = * oozie-site/oozie.authentication.kerberos.keytab = /etc/security/keytabs/oozie.ha.keytab
If this is followed in Ambari, when principals and keytab files are created (triggered by various events), Ambari will encounter errors since it will try to create a principle with the name of "*".
To avoid this problem, two Ambari-specific properties have been added and should be used to set the values specified above.
oozie-site/oozie.ha.authentication.kerberos.principal = * oozie-site/oozie.ha.authentication.kerberos.keytab = /etc/security/keytabs/oozie.ha.keytab
Notice the addition of ".ha" in the property names. This leave the real Oozie properties to be set as
oozie-site/oozie.authentication.kerberos.principal = HTTP/_HOST@${realm}
oozie-site/oozie.authentication.kerberos.keytab = /etc/security/keytabs/spngeo.service.keytabBecause the configurations used when creating principals and keytabs uses valid values, any operation the requires principals to be create will succeed as expected. Then, when the oozie-site.xml file is built by the Ambari agent, the agent-side logic will perform the proper replacement such that the following properties will be written to the file:
<property> <name>oozie.authentication.kerberos.principal</name> <value>*</value> </property> <property> <name>oozie.authentication.kerberos.keytab</name> <value>/etc/security/keytabs/oozie.ha.keytab</value> </property>
Steps to Follow
Given an Ambari-managed cluster where Kerberos is enabled and Oozie is installed, use the following steps to enable high availability for Oozie
1. Determine which host to use as the load balancer.
This host does not need to be part of the Ambari-managed cluster, but needs to be bidirectionally available to all host on it.
2. Install a load balancer on the select host.
For example, Pen, which is a simple light weight load balancer. For instructions on how to install Pen, see https://www.server-world.info/en/note?os=CentOS_6&p=pen.
If following instructions from the example above, the following changes should be made to /etc/pen.conf:
PORT needs to be changed from 80 to the port that Oozie is listening on. This is usually 11000.
PORT 11000
The SERVER entries need to point to the Oozie server instances (existing and to be installed)
SERVER1=<OOZIE_SERVER_HOST_1>:11000 SERVER2=<OOZIE_SERVER_HOST_2>:11000
3. Create the SPNEGO principal for host where the load balancer in installed
Creating the SPNEGO principal is only needed if the load balancer is installed on a host not already used in the Ambari-managed cluster. If the host already has services installed on it, Ambari should have previously created the needed principal.
The value of the SPNEGO principal is in the form of HTTP/<load balancer host FQDN>@<realm>. For example if the load balancer is installed on a host names load_balancer.example.com and the realm is EXAMPLE.COM, then the principal name is expected to be HTTP/local_balancer.example.com@EXAMPLE.COM.
Creating the principal is done differently depending on the type of KDC (or Active Directory) involved. If using an MIT KDC, the following command may be used:
kadmin -p <admin principal> -q 'add_principal -randkey HTTP/<load balancer host FQDN>@<realm>'
If issuing this command on the same host as the KDC as a user with appropriate privileges, then the following command may be used:
kadmin.local -q 'add_principal -randkey HTTP/<load balancer host FQDN>@<realm>'
4. Create the SPNEGO keytab file for host where the load balancer in installed
Creating the SPNEGO keytab file is only needed if the load balancer is installed on a host not already used in the Ambari-managed cluster. If the host already has services installed on it, Ambari should have previously created the needed keytab file.
Creating the principal is done differently depending on the type of KDC (or Active Directory) involved. If using an MIT KDC, the following command may be used:
kadmin -p <admin principal> -q 'xst -k lb.keytab HTTP/<load balancer host FQDN>@<realm>'
If issuing this command on the same host as the KDC as a user with appropriate privileges, then the following command may be used:
kadmin.local -q 'xst -k lb.keytab HTTP/<load balancer host FQDN>@<realm>'
It does not make a difference where the generated keytab file is stored; however, it should be protected again unauthorized access.
5. Create the Oozie HA keyab file
Gather the SPNEGO keytab files from the relevant hosts. This includes the host where the existing Oozie server is, the hosts where the new Oozie servers are to be installed, and the load balancer host. Care must be taken not to overwrite the different keytab files since most will have the name, spnego.service.keytab, but will contain a different set of keytabs.
One way to do this would be to crate a directory any copy the relevant keytab files into it using a unique name. For example, spnego.service.keytab.<source hostname>.
Once all files have been gathered, a composite keytab file may be created using the ktutil command line utility. This is done by executing ktutil on the command line, which invokes a shell. In the shell, a series of read_kt commands are to be issued (once for each keytab file), then a write_kt command is executed to create the composite keytab file. For example:
[root@host1 temp_keytabs]# ktutil ktutil: read_kt spnego.service.keytab.loadbalancer ktutil: read_kt spnego.service.keytab.host1 ktutil: read_kt spnego.service.keytab.host2 ktutil: write_kt oozie.ha.keytab ktutil: exit
Note, that care should be taken to protect access to the gathered keytab files. After the composite keytab file is properly distributed, the copied keytab file should be deleted.
6. Distribute the Oozie HA keytab file
The composite keytab file, ideally named oozie.ha.keytab, needs to be distributed to each Oozie server host. It does not need to be distributed to the load balancer host, unless that host contains an Oozie server instance.
The composite keytab files should be stored as /etc/security/keytabs/oozie.ha.keytab, or in whatever directory is used to store the keytab files. The access control for the file should be set such that only root and the designated Hadoop group (typically hadoop) has read access. For example:
chown root:hadoop /etc/security/keytabs/oozie.ha.keytab chmod 640 /etc/security/keytabs/oozie.ha.keytab
[root@host1 ~]# ls -l /etc/security/keytabs/oozie.ha.keytab -rw-r-----. 1 root hadoop 1434 Mar 15 23:38 /etc/security/keytabs/oozie.ha.keytab
7. Add the additional Oozie server instances
Log into Ambari as a user allowed to add new Oozie server instances. This is typically a user with Ambari Administrator, Cluster Administrator, or Cluster Operator privileges. Then browse to the view for each host where the new Oozie server instance is to be installed and select "Oozie Sever" from the component "Add" dropdown menu.
New Oozie server instances may not be automatically started. This is ok since the Oozie service will need to be stopped and started later.
8. Update the Oozie-specific configurations to enable HA
Log into Ambari as a user allowed to change service configurations. This is typically a user with Ambari Administrator, Cluster Administrator, Cluster Operator, or Service Administrator privileges. Then browse to the Oozie service configuration page and either add or update the following oozie-site properties. Adding a new oozie-site property may be done by clicking on the "Add Property ..." link under the "Custom oozie-site" section.
Add or update oozie.zookeeper.connection.string to contain the set of Zookeeper hosts and ports:
oozie.zookeeper.connection.string=<zookeeper_host_1>:2181,...,<zookeeper_host_n>:2181
Add or update oozie.services.ext:
oozie.services.ext=org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService
Set the Oozie server base URL to point to the load balancer:
oozie.base.url=http://<loadbalancer.hostname>:11000/oozie
Set the (Ambari-custom) Oozie HA-specific Kerberos principal and keytab properties:
oozie.ha.authentication.kerberos.principal=* oozie.ha.authentication.kerberos.keytab=/etc/security/keytabs/oozie.ha.keytab
Then, edit the oozie-env "oozie-env template" value
Uncomment the OOZIE_HTTP_HOSTNAME variable and set it to be the load balancer host
export OOZIE_HTTP_HOSTNAME=load_balancer.example.com
Ensure the following line is not commented out:
export OOZIE_BASE_URL="http://${OOZIE_HTTP_HOSTNAME}:${OOZIE_HTTP_PORT}/oozie"Finally, save the changes by clicking the "Save" button toward the top of the view and specifying a descriptive value like "Enabled Oozie HA".
9. Stop and start the Oozie service
Log into Ambari as a user allowed to start and stop services. This is typically a user with Ambari Administrator, Cluster Administrator, Cluster Operator, Service Administrator, or Service Operatorprivileges. Then browse to the Oozie service page and select "Stop" from the "Service Actions" dropdown. After the Oozie service successfully stops, select "Start" from the "Service Actions" dropdown.
Once the Oozie service has successfully started, test out the configuration by invoking the Oozie's service check. This is done by selecting "Run Service Check" from the "Service Actions" dropdown.
10. Locally test accessing Oozie via the load balancer using a Kerberos ticket
On any host in the Ambari-managed cluster
1. Login as a user that has a Kerberos identity. For example the Ambari smoke test user.
su - ambari-qa
2. Ensure a valid Kerberos ticket cache is established by kinit-ing
kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-c1
3. Issue a curl command to access Oozie (using "--negotiate -u:" to indicate Kerberos authentication is to be used)
curl -s -o /dev/null -k --negotiate -u: -w '%{http_code}\n' 'http://load_balancer.example.com:11000/oozie/?user.name=oozie'If 200 is returned, the test was a success. If 401 was returned, then there was an issue with Kerberos authentication.
Example:
[root@host1 ~]# su - ambari-qa
[ambari-qa@host1 ~]$ kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-c1
[ambari-qa@host1 ~]$ curl -s -o /dev/null -k --negotiate -u: -w '%{http_code}\n' 'http://load_balancer.example.com:11000/oozie/?user.name=oozie'
200
