Ranger Load Balancer URL not working after kerberization

[#1]

Hi,

I have provisioned an 8 node cluster with blueprint and enabled Ranger Admin HA. I am using haproxy for load-balancing.

The load-balancer URL works well when the cluster is not kerberized, however after Kerberization the load-balancer URL gives

503 Service Unavailable
No server is available to handle this request.

After kerberization, I referred the below HWX document for further steps(With SSL - Step 32 onwards)

Configure_ranger_admin_ha

After creating and copying the keytabs to the nodes, the URL still does not work.

Checked the namenode logs and found below error.

2018-04-03 09:56:44,677 INFO  ha.EditLogTailer (EditLogTailer.java:doTailEdits(275)) - Loaded 47 edits starting from txid 7367
2018-04-03 09:57:08,567 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
2018-04-03 09:57:38,570 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
2018-04-03 09:58:08,580 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
2018-04-03 09:58:38,587 WARN  client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop

I have also enabled haproxy logging but there is no error reported in the log.

Apr  4 08:21:39 localhost haproxy[927]: 10.13.1.30:39948 [04/Apr/2018:08:21:39.716] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763222321&pluginId=hiveServer2@hdp-mn01.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"
Apr  4 08:21:45 localhost haproxy[927]: 10.13.1.32:41364 [04/Apr/2018:08:21:45.512] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /login.jsp HTTP/1.1"
Apr  4 08:21:54 localhost haproxy[927]: 10.13.1.32:41426 [04/Apr/2018:08:21:54.915] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763235024&pluginId=hiveServer2@hdp-mn03.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"
Apr  4 08:21:58 localhost haproxy[927]: 10.13.1.30:40020 [04/Apr/2018:08:21:58.474] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /login.jsp HTTP/1.1"
Apr  4 08:22:00 localhost haproxy[927]: 10.13.1.32:41436 [04/Apr/2018:08:22:00.486] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hadoop?lastKnownVersion=4&lastActivationTime=1522768373827&pluginId=hdfs@hdp-mn03.clouddatadojo.com-hdpspark_hadoop&clusterName=hdpspark HTTP/1.1"
Apr  4 08:22:02 localhost haproxy[927]: 10.13.1.30:40052 [04/Apr/2018:08:22:02.071] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hadoop?lastKnownVersion=4&lastActivationTime=1522768375207&pluginId=hdfs@hdp-mn01.clouddatadojo.com-hdpspark_hadoop&clusterName=hdpspark HTTP/1.1"
Apr  4 08:22:09 localhost haproxy[927]: 10.13.1.30:40074 [04/Apr/2018:08:22:09.720] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763222321&pluginId=hiveServer2@hdp-mn01.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"

Hope you get the picture and can help out.

Thanks in advance,

Prabh

[#2]

Do you have SSL enabled for Ranger?

[#3]

hello , do you fix this problem ? i use httpd

and the warning is

@Prabhjot Singh

[#4]

Hi, when you enabling kerberos on cluster with Ranger HA enabled you need to:

1. Create service principals for load balancer nodes in AD or other Directory Manager (LB1,LB2,VIP);

2. Generate keytabs for this principals;

3. Merge generated keytabs with spnego.service.keytab;

You need to use hostnames for principals and LB VIP hostname for Ranger url

[#5]

@Giorgi Chitashvili, Still getting same issue. Could you please suggest?

[#6]

@Prabhjot Singh, I am also facing same issue. Did you get resolution of this issue? could you please suggest.

[#7]

I suggest you to follow this manual:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hadoop-high-availability/content/configu...

from 31 point and check all spn entries in spnego.service.keytab.

[#8]

@Giorgi Chitashvili, followed same steps. Still getting same issue. Could you please suggest?