Reply
Expert Contributor
Posts: 68
Registered: ‎10-04-2016
Accepted Solution

How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principals

I am on CDH 5.9.0 and using Cloudera Manager integrated with Active Directory to manage Kerberos ticket automatically. It is great until I am trying to enable Oozie HA via HAProxy.

 

How could I tell CM to generated HTTP keytab for oozie servers that contains HAProxy principal? I can do it manually. However, with CM Active Directory integration, I can't find a way to do so since I have no control of the keytab locations.

Cloudera Employee
Posts: 508
Registered: ‎07-30-2013

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

If you look at the Oozie config page, and search for load balancer, is that configured correctly?

Did you set up HA for Oozie using the CM wizard?

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_oozie_ha.html
Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

I checked oozie.keytab which has http principals for both proxy and local host, so the key tab is generated fine. However, Web UI "Load Balancer" gives me HTTP Status 403 - GSSException: Failure unspecified at GSS-API level - Checksum failed. However, both individual OOzie Web UIs return fine.

 

I am using HAProxy. The proxy URL worked fine before enabling Kerberos. Is there any specific setting I should do in HAProxy?

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

Yes. I enabled Oozie HA via CM.

Posts: 1,000
Topics: 1
Kudos: 249
Solutions: 126
Registered: ‎04-22-2014

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

You can check in Administration --> Security

Click on "Kerberos Credentials"

 

You can search for the hostname you entered as the proxy to view the credentials that are stored in Cloudera Manager

 

Cloudera Manager will automatically merge the keytabs and lay down the proper keytab in the oozie process directory at the time it is started.  You can do a klist on the file.  You can see the latest process directory by running:

 

ls -lrt /var/run/cloudera-scm-agent/process |grep OOZIE

 

 

-Ben

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

Ben,

CM did a good job on merging HTTP principals in oozie.keytab. However, my issue is the proxy. I got http 403 error on proxy UI, but not with two individual oozier server web UI.

Posts: 1,000
Topics: 1
Kudos: 249
Solutions: 126
Registered: ‎04-22-2014

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

Can you share the full error?

What is the URL you used to try to access the UI?

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

Web UI "Load Balancer" gives me HTTP Status 403 - GSSException: Failure unspecified at GSS-API level - Checksum failed. However, both individual OOzie Web UIs return fine.

Posts: 1,000
Topics: 1
Kudos: 249
Solutions: 126
Registered: ‎04-22-2014

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

This sounds more like a server-side exception.  I recommend checking the Oozie logs for exceptions being thrown when attempting to access the UI via load balancer.  The exception should hopefully shed some light on what is happening.

You could shut down one Oozie instance to ensure you know which log to look at.

 

 

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: How to merge oozie HA HTTP keberos principals with Cloudera Manager handling Kerboeros principal

Double-checked the KRB tickets, the principal for proxy is not using FQHN. I went back to check the LB configuration and sure it was using short name for the proxy host. Once I switched back, LB web UI comes back fine. Thanks.

Announcements