Create the self signed certificate and add it to a keystore file using: keytool -genkey -alias example.com -keyalg RSA -keystore keystore.jks -keysize 2048 2. List the keystore entries to verify that the certificate was added. Note that a keystore can contain multiple such certificates: keytool -list -keystore keystore.jks 3. Export this certificate from keystore.jks to a certificate file: keytool -export -alias example.com -file example.com.crt -keystore keystore.jks 4. Add this certificate to the client's truststore to establish trust: keytool -import -trustcacerts -alias example.com -file example.com.crt -keystore truststore.jks 5. Verify that the certificate exists in truststore.jks: keytool -list -keystore truststore.jks 6. Set hive.server2.thrift.sasl.qop=auth in HS2 configs Then start HiveServer2, login with user->kinit->beeline and try to connect with beeline using: !connect jdbc:hive2://<hs2_hostname>:10001/default;principal=<hive_principal>;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=<truststore_file_path>;trustStorePassword=<truststore_password> ... View more

SYMPTOM: Created a user in Ranger. User is visible in ranger DB but not visible in Ranger UI ERROR: Logged into mysql DB and executed below command - SELECT * FROM ranger.x_user where user_name in ('userA'); ==> This shows the user exist in x_user table. SELECT * FROM ranger.x_portal_user where user_name in ('userA'); ==> The user is also present in x_portal_user ROOT CAUSE: Suspected corruption on Ranger DB sometimes. RESOLUTION: Executing below command resolved the issue >INSERT INTO x_portal_user_role VALUES(NULL,'2016-09-09 00:00:00','2016-09-09 00:00:00',1,1,(SELECT id FROM x_portal_user WHERE login_id='XXXX'),'ROLE_USER',1); Replace XXXX with the login_id of the user ('userA') You can replace 'ROLE_USER' with 'ROLE_SYS_ADMIN' if you want it to be an admin ... View more

SYMPTOM: HDP upgrade was failed on HDFS startup. Namenode was not able to start and below were log messages - ERROR: From the detailed logs we see below error - ROOT CAUSE: The above log clearly indicates"ClassNotFound" error. Customer has integrated customer jar in hadoop which was causing the issue. RESOLUTION: There was custom jar which was already in place with Previous HDP version [located in path - /usr/hdp/2.4.3.0-37/hadoop/sas*.jar ]. Adding the jar from earlier version to the upgraded version path [ie. /usr/hdp/2.5.3.0-37/hadoop/] resolved the issue. [Note: There was custom implementation of SAS with hadoop for the setup and hence the custom jars were present in path mentioned above ie. /usr/hdp/2.4.3.0-37/hadoop/sas*.jar. Default setup never includes any custom app/jar implementation with hadoop. It usually refers or org.apache.hadoop class.] ... View more

SYMPTOM: Access Audit logs show 6 hours behind from Central Timezone. This is related to https://issues.apache.org/jira/browse/RANGER-336 ERROR: Ranger Access Audit logs show 6 hours behind from Central Timezone. ROOT CAUSE: This is bug - https://issues.apache.org/jira/browse/RANGER-336 RESOLUTION: Create a file name "ranger-admin-env-javaopts.sh" with below entry in path "/usr/hdp/current/ranger-admin/conf/" - export JAVA_OPTS=" ${JAVA_OPTS} -Duser.timezone=UTC" Save the file and restart Ranger admin service. ... View more

SYMPTOM: Ambari agent not able to register with Ambari server. ERROR: Below is the error logs - ERROR 2016-12-19 10:17:54,387 Controller.py:194 - Unable to connect to: https://oser402529.host.com:8441/agent/v1/register/oser402566.host.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 137, in registerWithServer data = json.dumps(self.register.build(self.version)) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 200, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 260, in iterencode return _iterencode(o, 0) UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 928: invalid start byte ERROR 2016-12-19 10:17:54,388 Controller.py:195 - Error:'utf8' codec can't decode byte 0xac in position 928: invalid start byte WARNING 2016-12-19 10:17:54,388 Controller.py:196 - Sleeping for 11 seconds and then trying again ERROR 2016-12-19 10:18:05,686 Controller.py:194 - Unable to connect to: https://oser402529.host.com:8441/agent/v1/register/oser402566.host.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 137, in registerWithServer data = json.dumps(self.register.build(self.version)) File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 230, in dumps return _default_encoder.encode(obj) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 200, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.6/site-packages/ambari_simplejson/encoder.py", line 260, in iterencode return _iterencode(o, 0) UnicodeDecodeError: 'utf8' codec can't decode byte 0xac in position 928: invalid start byte ROOT CAUSE: This is bug - https://hortonworks.jira.com/browse/BUG-52919 RESOLUTION: Create /usr/lib/python2.6/site-packages/sitecustomize.py file with the below content, restart ambari-agent. import sys sys.setdefaultencoding('utf-8’) Restart ambari-agent. ... View more

SYMPTOM: While performing step for Namenode HA, on the step of 'hdfs namenode -initializeSharedEdits' it failed with below error - ERROR: [root@localhost conf]# sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits' 16/11/22 09:43:56 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = locahost/10.200.206.12 STARTUP_MSG: args = [-initializeSharedEdits] STARTUP_MSG: version = 2.7.1.2.3.0.0-2557 STARTUP_MSG: classpath = /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/commons-cli-1.2.jar:/us r/hdp/2.3.0.0-2557/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.0.0-2557/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.3.0.0-2557/ . . . STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9f17d40a0f2046d217b2bff90ad6e2fc7e41f5e1; compiled by 'jenkins' on 2015-07-14T13:08Z STARTUP_MSG: java = 1.8.0_51 ************************************************************/ 16/11/22 09:43:56 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 16/11/22 09:43:56 INFO namenode.NameNode: createNameNode [-initializeSharedEdits] 16/11/22 09:43:56 ERROR namenode.NameNode: No shared edits directory configured for namespace null namenode null 16/11/22 09:43:56 INFO util.ExitUtil: Exiting with status 0 16/11/22 09:43:56 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at localhost/10.200.206.12 ************************************************************/ ROOT CAUSE: While performing NN Ha we found that Step 5 "Configure Components" is getting executed in a second, which was issue suspected. It was not stopping services and performing desired steps mentioned in screenshot below - RESOLUTION: Suspected that Ambari server was holding cache in DB. Clearing the Ambari server cache using below steps resolved the issue - # log out from the ambari server # list persisted k-v(ui cache) $curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/persist # get the cached state: $curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/persist/CLUSTER_CURRENT_STATUS # reset/clean the cache $curl -u admin:admin -H 'X-Requested-By:ambari' -d '{"CLUSTER_CURRENT_STATUS": "{\"clusterState\":\"DEFAULT\"}" }' -X POST 'http://localhost:8080/api/v1/persist' ... View more

Grafana username and password is stored in a sqlite3 database. One of the way is to reset the password back to admin first and then can be changed in Grafana Dashboard. To do the same, following steps could be used: 1. Logon to the node where Grafana is installed and invoke Grafana sqlite3 database as follows: # sqlite3 /var/lib/ambari-metrics-grafana/grafana.db sqlite> select salt, password from user; pyaUhfDzYg|54c7d1ce2eeaa6000bd84407d0f8ab4663dfa575e0a326bc70dc5cab4b864f6677b21879dbf5e33427c88f9160f744b625bf sqlite> update user set password = '59acf18b94d7eb0694c61e60ce44c110c7a683ac6a8f09580d626f90f4a242000746579358d77dd9e570e83fa24faa88a8a6', salt = 'F3FAxVm33R' where login = 'admin'; sqlite> .exit 2. Once done, edit Ambari Metrics Server-Configs and update Grafana Password to "admin" 3. Restart the Ambari Metrics Server 4. Access Grafana page using the Quick Links under Ambari Metric Server Dashboard 5. Click on the Grafana Symbol in the top left corner of the screen and Sign-in as admin user with "admin" password 6. Click on Global Users and edit admin user to change the password 7. Once the password is changed, change Grafana Admin password in Ambari Metrics Server-Configs as well Note: If the password is incorrect in Grafana database / Ambari Metrics Server configs, following errors would be observed: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana.py", line 64, in <module> AmsGrafana().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 535, in restart self.start(env) File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana.py", line 46, in start create_ams_datasource() File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana_util.py", line 230, in create_ams_datasource (response.status, response.reason, data)) resource_management.core.exceptions.Fail: Ambari Metrics Grafana data source creation failed. POST request status: 401 Unauthorized {"message":"Invalid username or password"} ... View more

Question: I have installed HDP cluster using Ambari which has Ranger service installed and working properly. I enabled the plugin Kafka for Ranger. I noticed something a little bit annoying. If Ranger Admin is down, Kafka will take a long time to start, because it tries to connect to Ranger Admin to get the repository. The error log is something like as shown below - ### Will retry 74 time(s), caught exception: Connection failed to Ranger Admin. Reason - [Errno 111] Connection refused.. Sleeping for 8 sec(s) ### Is there a way to decrease this number of retry or the duration of sleep between retry ? Ranger Admin down should not have any impact on the components for which the plugins are enabled, right ? Findings: Ambari uses the following Script to return the ranger admin login check response. /usr/lib/ambari-agent/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) /usr/lib/ambari-server/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) These scripts are having the hard coded values for the retry attempts and the sleep interval. something as following: {code} @safe_retry(times=75, sleep_time=8, backoff_factor=1, err_class=Fail, return_on_fail=None) def check_ranger_login_urllib2(self, url): """ :param url: ranger admin host url :param usernamepassword: user credentials using which repository needs to be searched. :return: Returns login check response """ . . . {code} So as a default behaviour Ambari will attempt total 75 times with sleeping interval of 8 seconds for the ranger admin login check response. If the ranger is down or if it does not comes up during these many times of attempt then it should throw the exception. REASON FOR ABOVE HARD CODED VALUES: 1. Blueprint based deployment as to make ensure the order of starting the services 2. Ranger admin startup can vary from environment to environment, hence the #retries was kept higher to be safe. HOW TO: Q. I would like to decrease this hardcoded values to one minute instead of 10 minutes. Which means 6 retries and 10s of sleep between retries. A. I see that Ambari uses the following Script to return the ranger admin login check response. #/usr/lib/ambari-agent/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) #/usr/lib/ambari-server/lib/resource_management/libraries/functions/ranger_functions_v2.py and (ranger_functions.py) The script 'ranger_functions_v2.py' (ranger_functions.py) controls these retry interval and sleep timing. Editing the scripts retry attempts and sleep can be a temporary suggestion. However altering the ambari provided scripts are not recommended without consulting Hortonworks. ... View more

PROBLEM STATEMENT: Ambari not displaying and service action buttons after login. It displays "Move Master Wizard In Progress". Not able to perform any operation from ambari and same is affecting for all users. ERROR: ROOT CAUSE: The issue seems that the user who logged inn to ambari UI[acting as admin], has tried to perform operation which was left in middle and just logged off from UI. The operation was cached in ambari UI and hence is reflected for other users in UI. RESOLUTION: Take ambari server backup and run below command from Ambari node - curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"<username>\",\"controllerName\":\"<controller_name>\"}"}' http://<ambari_host>:8080/api/v1/persist username = the user for which you are facing issue Ambari_host = hostname of ambari node controller_name = name of the controller for which ou are making request. curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"moveMasterController\"}"}'http://ambari.example.com:8080/api/v1/persist ... View more