Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ranger Load Balancer URL not working after kerberization

Ranger Load Balancer URL not working after kerberization

New Contributor

Hi,

I have provisioned an 8 node cluster with blueprint and enabled Ranger Admin HA. I am using haproxy for load-balancing.

The load-balancer URL works well when the cluster is not kerberized, however after Kerberization the load-balancer URL gives

503 Service Unavailable
No server is available to handle this request.

After kerberization, I referred the below HWX document for further steps(With SSL - Step 32 onwards)

Configure_ranger_admin_ha

After creating and copying the keytabs to the nodes, the URL still does not work.

Checked the namenode logs and found below error.

2018-04-03 09:56:44,677 INFO  ha.EditLogTailer (EditLogTailer.java:doTailEdits(275)) - Loaded 47 edits starting from txid 7367
2018-04-03 09:57:08,567 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
2018-04-03 09:57:38,570 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
2018-04-03 09:58:08,580 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
2018-04-03 09:58:38,587 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop

I have also enabled haproxy logging but there is no error reported in the log.

Apr  4 08:21:39 localhost haproxy[927]: 10.13.1.30:39948 [04/Apr/2018:08:21:39.716] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763222321&pluginId=hiveServer2@hdp-mn01.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"
Apr  4 08:21:45 localhost haproxy[927]: 10.13.1.32:41364 [04/Apr/2018:08:21:45.512] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /login.jsp HTTP/1.1"
Apr  4 08:21:54 localhost haproxy[927]: 10.13.1.32:41426 [04/Apr/2018:08:21:54.915] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763235024&pluginId=hiveServer2@hdp-mn03.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"
Apr  4 08:21:58 localhost haproxy[927]: 10.13.1.30:40020 [04/Apr/2018:08:21:58.474] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /login.jsp HTTP/1.1"
Apr  4 08:22:00 localhost haproxy[927]: 10.13.1.32:41436 [04/Apr/2018:08:22:00.486] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hadoop?lastKnownVersion=4&lastActivationTime=1522768373827&pluginId=hdfs@hdp-mn03.clouddatadojo.com-hdpspark_hadoop&clusterName=hdpspark HTTP/1.1"
Apr  4 08:22:02 localhost haproxy[927]: 10.13.1.30:40052 [04/Apr/2018:08:22:02.071] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hadoop?lastKnownVersion=4&lastActivationTime=1522768375207&pluginId=hdfs@hdp-mn01.clouddatadojo.com-hdpspark_hadoop&clusterName=hdpspark HTTP/1.1"
Apr  4 08:22:09 localhost haproxy[927]: 10.13.1.30:40074 [04/Apr/2018:08:22:09.720] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763222321&pluginId=hiveServer2@hdp-mn01.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"


Hope you get the picture and can help out.

Thanks in advance,

Prabh

7 REPLIES 7

Re: Ranger Load Balancer URL not working after kerberization

Do you have SSL enabled for Ranger?

Re: Ranger Load Balancer URL not working after kerberization

New Contributor

hello , do you fix this problem ? i use httpd

and the warning is

97595-1.jpg

@Prabhjot Singh

Re: Ranger Load Balancer URL not working after kerberization

New Contributor

Hi, when you enabling kerberos on cluster with Ranger HA enabled you need to:

1. Create service principals for load balancer nodes in AD or other Directory Manager (LB1,LB2,VIP);

2. Generate keytabs for this principals;

3. Merge generated keytabs with spnego.service.keytab;

You need to use hostnames for principals and LB VIP hostname for Ranger url

Highlighted

Re: Ranger Load Balancer URL not working after kerberization

Contributor

@Giorgi Chitashvili, Still getting same issue. Could you please suggest?

Re: Ranger Load Balancer URL not working after kerberization

Contributor

@Prabhjot Singh, I am also facing same issue. Did you get resolution of this issue? could you please suggest.

Re: Ranger Load Balancer URL not working after kerberization

New Contributor

I suggest you to follow this manual:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hadoop-high-availability/content/configu...

from 31 point and check all spn entries in spnego.service.keytab.

Re: Ranger Load Balancer URL not working after kerberization

Contributor

@Giorgi Chitashvili, followed same steps. Still getting same issue. Could you please suggest?