Created 04-04-2018 11:46 AM
Hi,
I have provisioned an 8 node cluster with blueprint and enabled Ranger Admin HA. I am using haproxy for load-balancing.
The load-balancer URL works well when the cluster is not kerberized, however after Kerberization the load-balancer URL gives
503 Service Unavailable No server is available to handle this request.
After kerberization, I referred the below HWX document for further steps(With SSL - Step 32 onwards)
After creating and copying the keytabs to the nodes, the URL still does not work.
Checked the namenode logs and found below error.
2018-04-03 09:56:44,677 INFO ha.EditLogTailer (EditLogTailer.java:doTailEdits(275)) - Loaded 47 edits starting from txid 7367 2018-04-03 09:57:08,567 WARN client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop 2018-04-03 09:57:38,570 WARN client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop 2018-04-03 09:58:08,580 WARN client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop 2018-04-03 09:58:38,587 WARN client.RangerAdminRESTClient (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(162)) - Error getting policies. secureMode=true, user=nn/hdp-mn01.clouddatadojo.com@WALGREENS.COM (auth:KERBEROS), response={"httpStatusCode":503,"statusCode":0}, serviceName=hdpspark_hadoop
I have also enabled haproxy logging but there is no error reported in the log.
Apr 4 08:21:39 localhost haproxy[927]: 10.13.1.30:39948 [04/Apr/2018:08:21:39.716] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763222321&pluginId=hiveServer2@hdp-mn01.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1" Apr 4 08:21:45 localhost haproxy[927]: 10.13.1.32:41364 [04/Apr/2018:08:21:45.512] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /login.jsp HTTP/1.1" Apr 4 08:21:54 localhost haproxy[927]: 10.13.1.32:41426 [04/Apr/2018:08:21:54.915] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763235024&pluginId=hiveServer2@hdp-mn03.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1" Apr 4 08:21:58 localhost haproxy[927]: 10.13.1.30:40020 [04/Apr/2018:08:21:58.474] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /login.jsp HTTP/1.1" Apr 4 08:22:00 localhost haproxy[927]: 10.13.1.32:41436 [04/Apr/2018:08:22:00.486] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hadoop?lastKnownVersion=4&lastActivationTime=1522768373827&pluginId=hdfs@hdp-mn03.clouddatadojo.com-hdpspark_hadoop&clusterName=hdpspark HTTP/1.1" Apr 4 08:22:02 localhost haproxy[927]: 10.13.1.30:40052 [04/Apr/2018:08:22:02.071] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hadoop?lastKnownVersion=4&lastActivationTime=1522768375207&pluginId=hdfs@hdp-mn01.clouddatadojo.com-hdpspark_hadoop&clusterName=hdpspark HTTP/1.1" Apr 4 08:22:09 localhost haproxy[927]: 10.13.1.30:40074 [04/Apr/2018:08:22:09.720] haproxy ranger_ha/<NOSRV> 0/-1/-1/-1/0 503 212 - - SCNN 0/0/0/0/0 0/0 "GET /service/plugins/secure/policies/download/hdpspark_hive?lastKnownVersion=6&lastActivationTime=1522763222321&pluginId=hiveServer2@hdp-mn01.clouddatadojo.com-hdpspark_hive&clusterName=hdpspark HTTP/1.1"
Hope you get the picture and can help out.
Thanks in advance,
Prabh
Created 04-27-2018 04:19 AM
Do you have SSL enabled for Ranger?
Created on 01-14-2019 03:28 PM - edited 08-17-2019 09:19 PM
Created 04-25-2022 12:05 AM
Has the problem been solved? I also have this problem and I look forward to your help
Created 01-15-2019 01:54 PM
Hi, when you enabling kerberos on cluster with Ranger HA enabled you need to:
1. Create service principals for load balancer nodes in AD or other Directory Manager (LB1,LB2,VIP);
2. Generate keytabs for this principals;
3. Merge generated keytabs with spnego.service.keytab;
You need to use hostnames for principals and LB VIP hostname for Ranger url
Created 01-29-2019 12:12 PM
@Giorgi Chitashvili, Still getting same issue. Could you please suggest?
Created 01-29-2019 12:11 PM
@Prabhjot Singh, I am also facing same issue. Did you get resolution of this issue? could you please suggest.
Created 01-29-2019 12:18 PM
I suggest you to follow this manual:
from 31 point and check all spn entries in spnego.service.keytab.
Created 01-29-2019 12:42 PM
@Giorgi Chitashvili, followed same steps. Still getting same issue. Could you please suggest?