Question: I have installed HDP cluster using Ambari which has Ranger service installed and working properly. I enabled the plugin Kafka for Ranger. I noticed something a little bit annoying. If Ranger Admin is down, Kafka will take a long time to start, because it tries to connect to Ranger Admin to get the repository. The error log is something like as shown below - ### Will retry 74 time(s), caught exception: Connection failed to Ranger Admin. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s) ### Is there a way to decrease this number of retry or the duration of sleep between retry ? Ranger Admin down should not have any impact on the components for which the plugins are enabled, right ? Findings: Ambari uses the following Script to return the ranger admin login check response. /usr/lib/ambari-agent/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) /usr/lib/ambari-server/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) These scripts are having the hard coded values for the retry attempts and the sleep interval. something as following: {code} @safe_retry(times=75, sleep_time=8, backoff_factor=1, err_class=Fail, return_on_fail=None) def check_ranger_login_urllib2(self, url): """ :param url: ranger admin host url :param usernamepassword: user credentials using which repository needs to be searched. :return: Returns login check response """ . . . {code} So as a default behaviour Ambari will attempt total 75 times with sleeping interval of 8 seconds for the ranger admin login check response. If the ranger is down or if it does not comes up during these many times of attempt then it should throw the exception. REASON FOR ABOVE HARD CODED VALUES: 1. Blueprint based deployment as to make ensure the order of starting the services 2. Ranger admin startup can vary from environment to environment, hence the #retries was kept higher to be safe. HOW TO: Q. I would like to decrease this hardcoded values to one minute instead of 10 minutes. Which means 6 retries and 10s of sleep between retries. A. I see that Ambari uses the following Script to return the ranger admin login check response. #/usr/lib/ambari-agent/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) #/usr/lib/ambari-server/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) The script 'ranger_functions_v2.py' (ranger_functions.py) controls these retry interval and sleep timing. Editing the scripts retry attempts and sleep can be a temporary suggestion. However altering the ambari provided scripts are not recommended without consulting Hortonworks. ... View more

PROBLEM STATEMENT: Ambari not displaying and service action buttons after login. It displays "Move Master Wizard In Progress". Not able to perform any operation from ambari and same is affecting for all users. ERROR: ROOT CAUSE: The issue seems that the user who logged inn to ambari UI[acting as admin], has tried to perform operation which was left in middle and just logged off from UI. The operation was cached in ambari UI and hence is reflected for other users in UI. RESOLUTION: Take ambari server backup and run below command from Ambari node - curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"<username>\",\"controllerName\":\"<controller_name>\"}"}' http://<ambari_host>:8080/api/v1/persist username = the user for which you are facing issue Ambari_host = hostname of ambari node controller_name = name of the controller for which ou are making request. curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"moveMasterController\"}"}'http://ambari.example.com:8080/api/v1/persist ... View more

PROBLEM STATEMENT: We have recently added 40 Datnodes added to cluster and they went down immediately after adding. Below was the exception found - ERROR: ROOT CAUSE: Customer added the 40nodes to the cluster running rack topology script as per link given - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_admin_tools/content/ch05.html The topology script which was copied by customer from above link was missing "fi" on last line and hence was giving above exception. RESOLUTION: Populated corrected script rack-topology.sh on namenode and datanodes and tried starting the datanodes after which datanode service was able to startup. ... View more

PROBLEM STATEMENT: Customer deleted hdfs repository from ranger ui. Tried re-enabling plugin but the repository was getting created. Reinstalled ranger, but still no luck. There was alert saying - "Ranger Admin Password check" Text is: This alert is used to ensure that the Ranger Admin password in Ambari is correct. Response is: User:amb_ranger_admin credentials on Ambari UI are not in sync with Ranger Doing a test and changing ranger_admin_password to admin in Ambari. Configs->Advanced Restarting Ranger, not prompted but doing so anyway. Also changed local password on ranger admin host: [root@ip-172-53-51-18 admin]# passwd amb_ranger_admin Changing password for user amb_ranger_admin. New password: Retype new password: passwd: all authentication tokens updated successfully. Did restart of HDFS, but now HDFS was not coming up. ERROR: Ranger admin log error: 2016-08-26 11:06:22,917 [http-bio-6080-exec-4] INFO org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:311) - Operation error. response=VXResponse={org.apache.ranger.view.VXResponse@7097c06cstatusCode={1} msgDesc={User is not allowed to update service-def, only Admin can create/update/delete Services} messageList={[VXMessage={org.apache.ranger.view.VXMessage@4718e9efname={OPER_NO_PERMISSION} rbKey={xa.error.oper_no_permission} message={User doesn't have permission to perform this operation} objectId={null} fieldName={null} }]} } From Ambari startup stderr box: 2016-08-26 11:06:20,803 - Error creating repository. Http status code - 400. {"statusCode":1,"msgDesc":"User is not allowed to update service-def, only Admin can create/update/delete Services","messageList":[{"name":"OPER_NO_PERMISSION","rbKey":"xa.error.oper_no_permission","message":"User doesn't have permission to perform this operation"}]} 2016-08-26 11:07:08,595 - Error creating repository. Http status code - 400. {"statusCode":1,"msgDesc":"User is not allowed to update service-def, only Admin can create/update/delete Services","messageList":[{"name":"OPER_NO_PERMISSION","rbKey":"xa.error.oper_no_permission","message":"User doesn't have permission to perform this operation"}]} 2016-08-26 11:07:56,368 - Error creating repository. Http status code - 400. {"statusCode":1,"msgDesc":"User is not allowed to update service-def, only Admin can create/update/delete Services","messageList":[{"name":"OPER_NO_PERMISSION","rbKey":"xa.error.oper_no_permission","message":"User doesn't have permission to perform this operation"}]} 2016-08-26 11:29:02,647 [http-bio-6080-exec-2] INFO org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:311) - Operation error. response=VXResponse={org.apache.ranger.view.VXResponse@6189b1d4statusCode={1} msgDesc={User is not allowed to update service-def, only Admin can create/update/delete Services} messageList={[VXMessage={org.apache.ranger.view.VXMessage@6291937cname={OPER_NO_PERMISSION} rbKey={xa.error.oper_no_permission} message={User doesn't have permission to perform this operation} objectId={null} fieldName={null} }]} } ROOT CAUSE: Role for 'amb_ranger_admin' user was not set to admin in ranger UI. RESOLUTION: Changing the role of 'amb_ranger_admin' user to admin from ranger UI resolved the issue. ... View more

PROBLEM STATEMENT: We have a strange problem with ranger. When I do a "select * from <table>; " I can see in ranger, on hive audit, that my user (dnid) is getting logged correctly. But when I look at the same situation on hdfs audit it shows that another user did the request. This Is to me very strange, I´ve tried with different users and the same problem happens again. ERROR: ROOT CAUSE: This is known issue and a BUG - https://issues.apache.org/jira/browse/HIVE-13120 http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_HDP_RelNotes/content/fixed_issues.html BUG-53108 HIVE-13120 RESOLUTION: Changed the below property in hiveserver2 configs and restarted hiveserver2 after which the Ranger HDFS audit was showing user as hive in HDFS audit. From: "hive.server2.enable.doAs"=true TO: "hive.server2.enable.doAs"=false ... View more

PROBLEM STATEMENT: There are frequent email alerts from HiveServer2 Metastore process which are misleading. When check the HiveServer2 Metastore process is up and running fine without issue. ERROR: Below is sample email alert - Ambari Alerts log output shows - /var/log/ambari-server/ambari-alerts.log ROOT CAUSE: The issue was with kerberos credentials for ambari-qa user. RESOLUTION: HiveServer2 Metastore alert is triggered from the node on which the service is installed. "ambari-qa" user is used by ambari to trigger the alert. If you are using kerberized cluster make sure you have "ambari-qa" ticket in place else false alert will be triggered. You can use below command to check - $su - ambari-qa $klist <--[make sure you have valid ticket] $hive --hiveconf hive.metastore.uris=thrift://<hive_mestastore_host>:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;' Also check, sometimes you might hit - https://issues.apache.org/jira/browse/AMBARI-14424 ... View more