ISSUE : Even after configuring falcon to use https, ambari alert checks Falcon UI on http and fails. Falcon service will be running fine but Ambari shows Falcon WEB UI alert. ROOT CAUSE : Ambari alert script is hardcoded to check Falcon on http WORKAROUND : 1. Login to Ambari UI 2. Open one more tab and go to this UI : http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/alert_definitions 3. Search for falcon_server_webui and get the alert definitions ID. Example : { "href" : "http://mon-ravi.ironman.com:8080/api/v1/clusters/ironman/alert_definitions/301", "AlertDefinition" : { "cluster_name" : "ironman", "id" : 301, "label" : "Falcon Server Web UI", "name" : "falcon_server_webui" } 301 is the alert definition here. 4. Delete this alert definition using curl curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/alert_definitions/301 5. On the ambari server, edit this file /var/lib/ambari-server/resources/common-services/FALCON/0.5.0.2.1/alerts.json Change "http": "{{falcon-env/falcon_port}}", to "https": "{{falcon-env/falcon_port}}", 6. Restart Ambari server We should be all good now. Falcon web UI alert should clear up. ... View more

ISSUE : In Ambari 2.4.x, Falcon Web UI is inaccessible(HTTP 503 error) and Ambari Service Check for Falcon fails: "ERROR: Unable to initialize Falcon Client object" Snippet of error from falcon logs 2017-01-04 14:08:56,390 INFO - [main:] ~ Initializing graph db (MetadataMappingService:143) 2017-01-04 14:08:56,438 ERROR - [main:] ~ Failed startup of context org.mortbay.jetty.webapp.WebAppContext@96a75da{/,/usr/hdp/current/falcon-server/server/webapp/falcon} (log:87) Caused by: org.apache.falcon.FalconException: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. Caused by: java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [com.thinkaurelius.titan.core.TitanFactory]. Caused by: java.lang.reflect.InvocationTargetException Caused by: java.lang.NoClassDefFoundError: com/sleepycat/je/LockMode Caused by: java.lang.ClassNotFoundException: com.sleepycat.je.LockMode 2017-01-04 14:08:56,440 INFO - [main:] ~ Started [email protected]:15000 (log:67) ROOT CAUSE : Missing jar WORK AROUND : On the falcon server, download missing jar and save it under falcon lib location wget http://search.maven.org/remotecontent?filepath=com/sleepycat/je/5.0.73/je-5.0.73.jar -O /usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/je-5.0.73.jar chown falcon:hadoop /usr/hdp/current/falcon-server/server/webapp/falcon/WEB-INF/lib/je-5.0.73.jar Restart Falcon server from Ambari. Hortonworks cannot ship this jar. It has to be manually downloaded. ... View more

ISSUE : If SOLR is configured as backend storage for ranger audits, ranger audit screen shows incorrect screen. This happens if server is running on any timezone other than UTC. ROOT CAUSE : 'ranger-admin-services.sh' script uses below command to start ranger. nohup java -Dproc_rangeradmin ${JAVA_OPTS} -Dlogdir=${RANGER_ADMIN_LOG_DIR} -Dcatalina.base=${XAPOLICYMGR_EWS_DIR} -cp "${XAPOLICYMGR_EWS_DIR}/webapp/WEB-INF/classes/conf:${XAPOLICYMGR_EWS_DIR}/lib/*:${RANGER_JAAS_LIB_DIR}/*:${RANGER_JAAS_CONF_DIR}:${JAVA_HOME}/lib/*:${RANGER_HADOOP_CONF_DIR}/*:$CLASSPATH" org.apache.ranger.server.tomcat.EmbeddedServer > ${RANGER_ADMIN_LOG_DIR}/catalina.out 2>&1 & If JAVA_OPTS value does not contain '-Duser.timezone=UTC', then Ranger admin runs in the local timezone. This causes timezone issue in viewing audit on Ranger UI as Ranger admin doesn't return eventtime in UTC format. RESOLUTION : This is fixed under https://issues.apache.org/jira/browse/RANGER-1249 WORKAROUND : 1. Create a file ranger-admin-env-javaopts.sh in /usr/hdp/current/ranger-admin/conf/ on the ranger admin server 2. Add the below contents in to the file. export JAVA_OPTS=" ${JAVA_OPTS} -Duser.timezone=UTC" 3. chown ranger:ranger /usr/hdp/current/ranger-admin/conf/ranger-admin-env-javaopts.sh 4. chmod 755 /usr/hdp/current/ranger-admin/conf/ranger-admin-env-javaopts.sh 5. Restart Ranger admin. ... View more

ISSUE : Ranger HA fails if admin password is changed to non default password Error stack : Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 208, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 59, in install setup_ranger_admin_passwd_change() File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 598, in setup_ranger_admin_passwd_change Execute(cmd, environment={'JAVA_HOME': params.java_home, 'RANGER_ADMIN_HOME': params.ranger_home}, user=params.unix_user) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'ambari-python-wrap /usr/hdp/current/ranger-admin/db_setup.py -changepassword admin [PROTECTED] [PROTECTED]' returned 1. 2016-12-02 19:34:37,314 [I] DB FLAVOR :MYSQL 2016-12-02 19:34:37,314 [I] --------- Verifying Ranger DB connection --------- 2016-12-02 19:34:37,315 [I] Checking connection.. 2016-12-02 19:34:37,315 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "SELECT version();" 2016-12-02 19:34:37,808 [I] Checking connection passed. 2016-12-02 19:34:37,809 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DEFAULT_ADMIN_UPDATE' and active = 'Y';" 2016-12-02 19:34:38,268 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DEFAULT_ADMIN_UPDATE' and active = 'N';" 2016-12-02 19:34:38,727 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('DEFAULT_ADMIN_UPDATE', now(), user(), now(), user(),'N') ;" 2016-12-02 19:34:39,192 [I] Ranger admin default password change request is in process.. Invalid user password 2016-12-02 19:34:49,401 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='DEFAULT_ADMIN_UPDATE' and active='N';" 2016-12-02 19:34:49,865 [E] Ranger admin default password change request failed ROOT CAUSE : This is product defect and engineering are working on a fix. RESOLUTION / WORKAROUND : Set the ranger admin password to " admin " in both Ranger UI and Ambari -> Ranger Configs and retrigger Ranger HA installation. Once installation is complete, we can change the ranger admin password. ... View more

ISSUE : Ranger HA fails if admin password is changed to non default password Error stack : Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 208, in <module> RangerAdmin().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 59, in install setup_ranger_admin_passwd_change() File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 598, in setup_ranger_admin_passwd_change Execute(cmd, environment={'JAVA_HOME': params.java_home, 'RANGER_ADMIN_HOME': params.ranger_home}, user=params.unix_user) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'ambari-python-wrap /usr/hdp/current/ranger-admin/db_setup.py -changepassword admin [PROTECTED] [PROTECTED]' returned 1. 2016-12-02 19:34:37,314 [I] DB FLAVOR :MYSQL 2016-12-02 19:34:37,314 [I] --------- Verifying Ranger DB connection --------- 2016-12-02 19:34:37,315 [I] Checking connection.. 2016-12-02 19:34:37,315 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "SELECT version();" 2016-12-02 19:34:37,808 [I] Checking connection passed. 2016-12-02 19:34:37,809 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DEFAULT_ADMIN_UPDATE' and active = 'Y';" 2016-12-02 19:34:38,268 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DEFAULT_ADMIN_UPDATE' and active = 'N';" 2016-12-02 19:34:38,727 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('DEFAULT_ADMIN_UPDATE', now(), user(), now(), user(),'N') ;" 2016-12-02 19:34:39,192 [I] Ranger admin default password change request is in process.. Invalid user password 2016-12-02 19:34:49,401 [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://gwy-ravi.ironman.com/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='DEFAULT_ADMIN_UPDATE' and active='N';" 2016-12-02 19:34:49,865 [E] Ranger admin default password change request failed ROOT CAUSE : This is product defect and engineering are working on a fix RESOLUTION / WORKAROUND : Set the ranger admin password to " admin " in both Ranger UI and Ambari -> Ranger Configs and retrigger Ranger HA installation. Once installation is complete, we can change the ranger admin password. ... View more

ISSUE : When we click on Datanodes tab on Namenode UI, nothing happens. ROOT CAUSE : This is happening because of https://issues.apache.org/jira/browse/HDFS-10888 and https://issues.apache.org/jira/browse/HDFS-9193 which is fixed in Hadoop version 2.8. But HDP 2.5 is on Hadoop 2.7. RESOLUTION/WORKAROUND : To resolve this, we can work around by editing the below file on both the namenodes /usr/hdp/current/hadoop-hdfs-client/webapps/hdfs/dfshealth.js and changing } else if (u.usedPercentage < 85) { to } else if (v < 85) { We should be able to access datanode tabs after restarting namenode. ... View more