question Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal in Archives of Support Questions (Read Only)

How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principals

zhuw.bigdata — Fri, 16 Sep 2022 10:51:53 GMT

I am on CDH 5.9.0 and using Cloudera Manager integrated with Active Directory to manage Kerberos ticket automatically. It is great until I am trying to enable Oozie HA via HAProxy.

How could I tell CM to generated HTTP keytab for oozie servers that contains HAProxy principal? I can do it manually. However, with CM Active Directory integration, I can't find a way to do so since I have no control of the keytab locations.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

Darren — Wed, 21 Dec 2016 18:38:02 GMT

If you look at the Oozie config page, and search for load balancer, is that configured correctly?

Did you set up HA for Oozie using the CM wizard?

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_oozie_ha.html

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

zhuw.bigdata — Wed, 21 Dec 2016 20:25:40 GMT

I checked oozie.keytab which has http principals for both proxy and local host, so the key tab is generated fine. However, Web UI "Load Balancer" gives me HTTP Status 403 - GSSException: Failure unspecified at GSS-API level - Checksum failed. However, both individual OOzie Web UIs return fine.

I am using HAProxy. The proxy URL worked fine before enabling Kerberos. Is there any specific setting I should do in HAProxy?

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

zhuw.bigdata — Wed, 21 Dec 2016 20:26:19 GMT

Yes. I enabled Oozie HA via CM.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

bgooley — Wed, 21 Dec 2016 20:46:04 GMT

You can check in Administration --> Security

Click on "Kerberos Credentials"

You can search for the hostname you entered as the proxy to view the credentials that are stored in Cloudera Manager

Cloudera Manager will automatically merge the keytabs and lay down the proper keytab in the oozie process directory at the time it is started. You can do a klist on the file. You can see the latest process directory by running:

ls -lrt /var/run/cloudera-scm-agent/process |grep OOZIE

-Ben

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

zhuw.bigdata — Wed, 21 Dec 2016 20:52:14 GMT

Ben,

CM did a good job on merging HTTP principals in oozie.keytab. However, my issue is the proxy. I got http 403 error on proxy UI, but not with two individual oozier server web UI.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

bgooley — Wed, 21 Dec 2016 21:02:43 GMT

Can you share the full error?

What is the URL you used to try to access the UI?

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

zhuw.bigdata — Wed, 21 Dec 2016 21:19:14 GMT

Web UI "Load Balancer" gives me HTTP Status 403 - GSSException: Failure unspecified at GSS-API level - Checksum failed. However, both individual OOzie Web UIs return fine.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

bgooley — Wed, 21 Dec 2016 21:38:47 GMT

This sounds more like a server-side exception. I recommend checking the Oozie logs for exceptions being thrown when attempting to access the UI via load balancer. The exception should hopefully shed some light on what is happening.

You could shut down one Oozie instance to ensure you know which log to look at.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

zhuw.bigdata — Wed, 21 Dec 2016 21:54:31 GMT

Double-checked the KRB tickets, the principal for proxy is not using FQHN. I went back to check the LB configuration and sure it was using short name for the proxy host. Once I switched back, LB web UI comes back fine. Thanks.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

bgooley — Wed, 21 Dec 2016 23:36:45 GMT

That's great! Nice detective work.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

zhuw.bigdata — Fri, 23 Dec 2016 20:04:31 GMT

Here is the description for the LB:

Address of the load balancer used if Oozie HA is enabled. Should be specified in host:port format.

Could we improve it to "FQHN:port" and ask technical writer to update it? Kerberos only treats FQHN well.

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

bgooley — Thu, 29 Dec 2016 18:22:57 GMT

@zhuw.bigdata, I opened two internal Cloudera Jiras to make sure we specify that the fully-qualified domain name be used if Kerberos is enabled in the cluster. One Jira targeted the description in the HA wizard, the other Jira focused on the steps listed in our the documentation.

Thanks for bringing this up!

Cheers,

Ben