Resources

SYMPTOM: Cluster was upgraded to 2.3 After the upgrade oozie has configuration issues. User has workflow defined to create job files in directory /tmp/hadoop-${user.name}/job_details but instead, the directory is getting created in / and permission denied error is thrown ERROR: Sample workflow:<workflow-app xmlns='uri:oozie:workflow:0.5' name='scisit_all_oozie_workflow'> <global> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <job-xml>${runtime}/runtime_params.xml</job-xml> <job-xml>scisit_all_tables_config.xml</job-xml> <job-xml>ColumnTransformationRules.xml</job-xml> <job-xml>HeadersAndTrailers.xml</job-xml> <configuration> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>oozie.action.sharelib.for.java</name> <value>hive</value&gt

SYMPTOM: A create table in a cluster enforcing authorization using Ranger, fails to create the table and post that HiveServer2 process crashes. Create table fails as follows 0: jdbc:hive2://xxxx.hk.example.com> CREATE EXTERNAL TABLE TMP_HIVE2PHOENIX_E32E8 (CUSTOMER_ID STRING, ACCOUNT_ID STRING, ROLE_ID STRING, ROLE_NAME STRING, START_DATE STRING, END_DATE STRING, PRIORITY STRING, ACTIVE_ACCOUNT_ROLE STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/tmp/example' TBLPROPERTIES ('serialization.null.format'=''); Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0) Following errors are observed in hiveserver2.log: 2016-11-15 11:42:06,721 WARN [HiveServer2-Handler-Pool: Thread-32350]: thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(492)) - Error executing statement:org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission den

SYMPTOM: - Enabled Ranger Kafka plugin via Ambari and restarted kafka service.- kafka logs still populating with "Ranger Plugin returned null" error - check ranger logs and could not see any info about kafka policy download - checked /etc/ranger/test_kafka/policycache/ and the json file in that is empty! bash-4.1# cd /etc/ranger/test_kafka/ bash-4.1# cd policycache/ bash-4.1# ls -ltr total 0 -rw-r--r-- 1 kafka hadoop 0 Mar 2 16:00 kafka_test_kafka.json_old -rw-r--r-- 1 kafka hadoop 0 Mar 16 11:30 kafka_test_kafka.json_old1 -rw-r--r-- 1 kafka hadoop 0 Mar 16 12:27 kafka_test_kafka.json - checked Test Connection for Kafka repo in Ranger. It was successful. - The Ranger plugin audits, did not have info on the kafka plugin sync. - Thus the kafka plugin is not being synced in this case. Policy refresh not working. - Tried deleting the default kafka policy and created a new one however issue still exists. - Tried to use REST API to get the policy details however no output. ERROR:2016-0

SYMPTOM: All the services in the cluster are down and restarting the services fails with the following error: 2016-11-17 21:42:18,235 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.io.IOException: Login failure for nn/lxxx.examplet.ex.com@EXAMPLE.AD.EX.COM from keytab /etc/security/keytabs/nn.service.keytab: javax.security.auth.login.LoginException: Client not found in Kerberos database (6) ...Caused by: KrbException: Client not found in Kerberos database (6) ...Caused by: KrbException: Identifier doesn't match expected value (906) Regeneration of Keytabs using Ambari too failed as follows: 17 Nov 2016 23:58:59,136 WARN [Server Action Executor Worker 12702] CreatePrincipalsServerAction:233 - Principal, HTTP/xxx.examplet.ex.com@EXAMPLE.AD.EX.COM, does not exist, creating new principal 17 Nov 2016 23:58:59,151 ERROR [Server Action Executor Worker 12702] CreatePrincipalsServerAction:284 - Failed to create or update principal, HTTP/xxx.examplet.ex.com@E

SYMPTOMS: Due to both Resource manager getting active simultaneously , all node managers crash. Errors visible in RM logs are as follows:2015-06-27 20:08:35,922 DEBUG [main] service.AbstractService (AbstractService.java:enterState(452)) - Service: Dispatcher entered state STOPPED2015-06-27 20:08:35,923 WARN [AsyncDispatcher event handler] event.AsyncDispatcher (AsyncDispatcher.java:handle(247)) - AsyncDispatcher thread interruptedjava.lang.InterruptedExceptionat java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)atjava.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)at org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:244)at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.updateApplicationAttemptState(RMStateStore.java:652)at org.apache.hadoop.yarn.server.res

SYMPTOMSometimes while performing any Ambari operation like for example, adding a Ranger KMS service or doing any other operation, the Ambari UI (or even curt response) might show this error:ERROR:Error 500 status code received on GET method for API: /api/v1/stacks/HDP/versions/2.3/recommendationsError message: Error occurred during stack advisor command invocation: Cannot create /var/run/ambari-server/stack-recommendationsROOT CAUSE This is most likely a permission issue with /var/run/ambari-server/ location for the user who is running the 'ambari-server' process.RESOLUTION To resolve this, the permission for /var/run/ambari/server should be setup correctly. On the good cluster, where the ambari-server is running as 'root' the permission looks like: [root@test ~]# ll -d /var/run/ambari-server/drwxr-xr-x 4 root root 4096 Dec 4 05:10 /var/run/ambari-server/[root@test ~]# ll /var/run/ambari-server/total 12-rw-r--r-- 1 root root 6 Dec 4 05:10 ambari-server.piddrwxr-xr-x 4 root root 4

PROBLEM: After upgrading from Ambari from v2.1.2.1 to v2.2.2.0 when attempting to "Re-install" Grafana nothing happened. No task was started and the Ambari UI seemed to hang.ERROR: The following error messages were found in the Ambari server log file08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=kerberos-env 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=krb5-conf 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=ranger-ugsync-site 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=admin-properties 08 Jul 2016 10:28:38,174 ERROR [qtp-ambari-client-2410] ClusterImpl:2347 - Config inconsistency exists: unknown configType=usersync-properties 08 Jul 2016 10:28:38,175 ERROR [qtp-ambari-client-2410]

SYMPTOM: When trying to enable the Ranger plugin for the HIVE component and then clicking SAVE, it does not save this as a new configuration.Certain Hive Smart Config changes cannot be made (set Authorization to Ranger, etc) when Oracle JDBC URL contains a non-standard port. ROOT CAUSE: https://hortonworks.jira.com/browse/BUG-50133 WORKAROUND: Reach out to Hortonworks Support for a hotfix patch and instructions to address this issue.

SYMPTOM: Nagios install fails due to existing "nagios" user in the ldap systemtrying to install ambari on single node cluster but is having issues with nagios since they have an ldap server with a nagios user already created.ERROR: Receiving error; "err: /Stage2/Hdp-nagios::Server/Hdp::Usernagios/Usernagios/gid: change from 20000 to nagios failed: Could not set gid on usernagios: Execution of '/usr/sbin/usermod -g 492 nagios' returned 6: usermod: user 'nagios' does not exist in /etc/passwd ROOT CAUSE: This is BUG - https://hortonworks.jira.com/browse/BUG-6787 RESOLUTION / WORKAROUND:Workaround #1: That user must be in group = nagios as well, or the nagios install fails. The workaround for this is to add nagios user to a nagios group.Workaround #2: During install, customize nagios user (Customize Services > Misc) and use something other than nagios user. This user will be created and put in a group nagios. This is available in HDP 1.3.1

Problem Description: often we are in need to read and write underlying files from user defined reader and writer. if the the custom reader and writer are written in java or language run over JVM then we are good to add those in hive_aux_jar or we can add them using add jar option at session level but when if it is written in native language and shipped as *.so file then we will get java.lang.UnsatisfiedLinkError. we can workaround this problem after adding it to hive-env 1. open Ambari-->Hive-->Advanced-->Advanced hive-env-->hive-env template 2. modify {% if sqla_db_used or lib_dir_available %} export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:{{jdbc_libs_dir}}" export JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:{{jdbc_libs_dir}}" {% endif %}

SYMPTOM: External group synced from Unix/LDAP/AD system are labelled as ‘Internal’ in Ranger Admin UI.Currently, when I check under Ranger -> Settings -> Groups, I see around 1700 groups. And all the groups are shown as internal. However, when I login to the Linux hosts I see only 77 groups. I am wondering how to identify from UI which groups are internal and external as all the groups are shown as internal. ERROR: 1. After Ranger and Usersync installation, started Ranger admin and Usersync to sync UNIX OS users. 2. Logged in to Ranger Admin Dashboard and visited User/Group section, then clicked on Group tab.External group synced from Unix/LDAP/AD system are labelled as ‘Internal’ in Ranger Admin UI. ROOT CAUSE: This bug is targeted for inclusion in the Release Notes>Known Issue in HDP 2.3.4, as it has fixVersion in (Dal-M21, Dal-M30, DAL-M30, Dal-Next)https://hortonworks.jira.com/browse/BUG-47662 [HWX internal url for tracking purpose] https://reviews.apache.org/r/42146/ RE

Question: We would like to Understand the mechanism behind the properties yarn.scheduler.capacity.node-locality-delay==> All the YARN schedulers try to honor locality requests. On a busy cluster, if an application requests a particular node, there is a good chance that other containers are running on it at the time of the request. The obvious course of action is to immediately loosen the locality requirement and allocate a container on the same rack. However, it has been observed in practice that waiting a short time (no more than a few seconds) can dramatically increase the chances of being allocated a container on the requested node, and therefore increase the efficiency of the cluster. This feature is called delay scheduling, and it is supported by both the Capacity Scheduler and the Fair Scheduler. Every node manager in a YARN cluster periodically sends a heartbeat request to the resource manager—by default, one per second. Heartbeats carry information about the node manager’s

SYMPTOM: While following steps for HDP pre-upgrade activity the active namenode went down while issuing - #hdfs dfsadmin -savenamespace" command.Below was the error -================================================ hdfs@namenode1~$ hdfs dfsadmin -saveNamespace saveNamespace: Call From namenode1.example.com/10.160.81.30 to namenode1.example.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused================================================ERROR: 2016-06-14 02:18:49,774 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(209)) - Transport-level exception trying to monitor health of NameNode at namenode1.example.com/10.10.20.30:8020: Call From namenode1.example.com/10.10.20.30 to namenode1.example.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused2016-06-14 02:18:51,774 INF

SYMPTOM : While installing Ranger through Ambari , on Oracle DB, user runs into error "sqlplus64: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory" ERROR:sqlplus64: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directoryROOT CAUSE : possible missing environment settings and misconfiguration in Ambari .RESOLUTION : 1. On Ambari Server Machine, please check environment variable in Linux OS level, verify environmental path LD_LIBRARY_PATH=/usr/lib/oracle/<version>/client64/lib is set. 2. AND in Ambari config -> ranger config - advanced ranger-env, set property "oracle_home" with "/usr/lib/oracle/<version>/client64/lib"

Ambari Server Performance - This alert appears when the size of the below API call does not respond back within the threshold limit configured in the alert notifications. The reason for this could be the content of the API response just got bigger over the releases. curl: http://<host>:<port>api/v1/clusters/<clustername>; Ambari alert sample: Alert 1: (Time : 01 December 2016 19:42) CRITICAL Ambari Server Performance Performance Overview: Database Access (Request By Status): 1ms (OK) Database Access (Task Status Aggregation): 1ms (OK) REST API (Cluster): 16,454ms (CRITICAL) Cluster: <clustername> Alert2 : (Time : 02 December 2016 10:40 ) Ambari Server Performance Performance Overview: Database Access (Request By Status): 1ms (OK) Database Access (Task Status Aggregation): 1ms (OK) REST API (Cluster): 8,544ms (CRITICAL) Cluster: <clustername> Possible fix: Increase the threshold limit configured in the alert notification or disable the alert. In future r

SYMPTOM: During an upgrade to Ambari 2.1.2 and HDP 2.3.2 the upgrade fails when trying to start the namenode service. Errors in the stack trace indicate that the xasecure files are not found.ISSUE: When Ranger has been installed outside of Ambari, the method used for previous HDP versions, the Ambari database does not contain information about Ranger. As a result the configuration files are not replicated to the new directory structure, /etc/hadoop/<version>/0/. Since the Ranger properties are included in other component configuration files the namenode service will not start. A result of the missing configuration files for Ranger.RESOLUTION: If Ranger was installed outside of Ambari you can replicate the xasecure files from the previous version to the current version configuration directory. cp /etc/hadoop/conf/xasecure* /etc/hadoop/<version>/0/ ie. cp /etc/hadoop/conf/xasecure* /etc/hadoop/2.3.2.0-2950/0/ ** You will need to complete this on all namenode servers in t

SYMPTOMS: Although valid Kerberos ticket is available, we are unable to put files in HDFS encrypted zone. If we restart/failover namenode, then we are able to put files with the same ticket/credentials without having to get a new ticket. Below is the demo of the issue where /tmp/user1 is the encrypted zone and user has permission to that zone: [root@test ~]# su - user Last login: Thu Oct 13 13:03:24 EDT 2016 on pts/57 -bash-4.2$ id uid=11516(user) gid=5000(bns) groups=5000(bns),1520(cmtsuser),1800(admin),4534(edgegrp),4535(edgedgrp),4536(k2tstgrp),8242(ftallocctxd),8243(ftallocctxu),15113(hdpadm) -bash-4.2$ kinit Password for user@123.EXAMPLE.COM: -bash-4.2$ klist Ticket cache: FILE:/tmp/krb5cc_11516 Default principal: user@123.EXAMPLE.COMValid starting Expires Service principal 10/14/2016 07:23:51 10/14/2016 17:23:51 krbtgt/123.EXAMPLE.COM@EXAMPLE.COM renew until 10/21/2016 07:23:48 -bash-4.2$ hadoop fs -put file1 /tmp/user1/file_1 put: java.util.concurrent.ExecutionException: java.io

SYMPTOMS: When local disk utilization of multiple node managers goes high beyond a limit, nodes turn “unhealthy” and gets into the "blacklist" not to be used for container/task allocation, hence reducing the effective cluster capacity.ROOT CAUSE: A burst or rapid rate of submitted jobs with substantial NM usercache resource localization footprint may lead to rapid fill up of the NM local temporary file system with negative consequences in terms of stability. The core issue seems to be the fact that NM continues to localize the resources beyond the maximum local cache size (yarn.nodemanager.localizer.cache.target-size-mb , default 10G). Since maximum local cache size is effectively not taken into account when localizing new resources (note that default cache cleanup interval is 10 min controlled by yarn.nodemanager.localizer.cache.cleanup.interval-ms), this basically leads to sort of self-destruction scenario : Once the filesystem utilization reaches the threshold of 90%, NM will autom

SYMPTOM: We had problem with namenode starts. We can check if there is namenode startup issue then we can check for gc related errors in logs and same time check for list of files/blocks customer is having on cluster and the recommended HEAP size as per hortonworks manual - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html ROOT CAUSE: GC errors were causing the issue for namenode to startRESOLUTION: Below are the steps taken to resolve the issue -1. Logged in to the namenode cli2. When checked from cli using “ps –ref |grep –i namenode”, the namenode was not displaying. 3. Seems that the namenode process was getting killed after specific interval of time, but ambari was still showing the namenode process state in UI as “starting” 4. Cancelled the namenode starting process from Ambari UI. 5. Tried starting the whole HDFS process and simultaneously ran “IOSTAT” on the fsimage disk. 6, We found withi

Problem Statement: when doing an ambari-server sync-ldap -groups=<your file>It will bring over the groups but not the users in it. ROOT CAUSE: When troubleshooting why the group members are not being sync'd with FreeIPA, a packet trace helped identify the issue. With ActiveDirectory the user's DN is exposed as an attribute: "distinguishedName", this is not the case inFreeIPA/RHEL IDM (using 389 DS for the directory server implementation). The DN is not an attribute on the user, and cannot be used in a filter like this:(&(objectClass=posixaccount)(|(dn=uid=dstreev,cn=users,cn=accounts,dc=hdp,dc=local)(uid=uid=dstreev,cn=users,cn=accounts,dc=hdp,dc=local)))If we want to retrieve a specific object by DN we have to set the DN as the search base and do a base search scope.ldapsearch -H ldap://ad.hortonworks.local:389 -x -D "CN=hadoopsvc,CN=Users,dc=hortonworks,dc=local" -W -b "CN=paul,CN=Users,DC=hortonworks,DC=local" -s base -a always "(objectClass=user)"In this case I'm looking

SYMPTOM: oozie-hive job is running very slow, sometimes jobs are stuck is in final stage and was not able to completeROOT CAUSE:Oozie prepares the hive-site.xml for hive action, it has the mapred parameter mapreduce.job.reduces set to 1 by default,the reason for this is oozie prepare action configuration after reading core-site.xml,hdfs-site.xml,mapred-site.xml etc.with the setting of mapreduce.job.reduces=1 the job is running with single reducer hence taking a lot of time to complete.WORKAROUND:set mapreduce.job.reduces to -1RESOLUTION:there is oozie fix https://issues.apache.org/jira/browse/OOZIE-2205 enhance actionConf which are passed to hive-action.

SYMPTOM: When Ambari is upgraded to 2.4.x, SmartSense has to be upgraded before upgrading the HDP stack.Failing to do so, would cause SmartSense to capture bundle with small size (in KB's) that has very less data.ERROR: ERROR 2016-12-01 23:02:02,279 anonymize.py:63 - Execution of command returned 1. Exception in thread "main" java.lang.NullPointerException at com.hortonworks.smartsense.anonymization.rules.RuleType.from(RuleType.java:47) at com.hortonworks.smartsense.anonymization.rules.Rules.createRule(Rules.java:161) at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:148) at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:132) at com.hortonworks.smartsense.anonymization.AnonymizerFactory.createAnonymizer(AnonymizerFactory.java:94) at com.hortonworks.smartsense.anonymization.Main.start(Main.java:202) at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294)ERROR 2016-12-01 23:02:02,280 AnonymizeBundleCommand.py:60 - Ano

ISSUE : If SOLR is configured as backend storage for ranger audits, ranger audit screen shows incorrect screen. This happens if server is running on any timezone other than UTC.ROOT CAUSE :'ranger-admin-services.sh' script uses below command to start ranger.nohup java -Dproc_rangeradmin ${JAVA_OPTS} -Dlogdir=${RANGER_ADMIN_LOG_DIR} -Dcatalina.base=${XAPOLICYMGR_EWS_DIR} -cp "${XAPOLICYMGR_EWS_DIR}/webapp/WEB-INF/classes/conf:${XAPOLICYMGR_EWS_DIR}/lib/*:${RANGER_JAAS_LIB_DIR}/*:${RANGER_JAAS_CONF_DIR}:${JAVA_HOME}/lib/*:${RANGER_HADOOP_CONF_DIR}/*:$CLASSPATH" org.apache.ranger.server.tomcat.EmbeddedServer > ${RANGER_ADMIN_LOG_DIR}/catalina.out 2>&1 &If JAVA_OPTS value does not contain '-Duser.timezone=UTC', then Ranger admin runs in the local timezone. This causes timezone issue in viewing audit on Ranger UI as Ranger admin doesn't return eventtime in UTC format.RESOLUTION : This is fixed under https://issues.apache.org/jira/browse/RANGER-1249WORKAROUND :1. Create a file

SYMPTOM: We are connected to hue using hdfs user. After executing oozie w/f using HUE the workflow is failing.Below is sample job definition -<workflow-app name="Headlight" xmlns="uri:oozie:workflow:0.4"> <start to="Headlight"/> <action name="Headlight"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <exec>/user/hdfs/oozie/deployments/headlight-exec-all.pl</exec> </shell> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> ERROR: Tried creating simple script and ran, but the files are created using yarn user as shown below -#!/bin/sh /bin/touch /tmp/test.oozie The result is ok. the file is created by the user yarn. -rw-r--r-- 1 yarn hadoop 0 Jun 3 03:04 /tmp/

ENVIRONMENT: HDP 2.3.4, Ambari 2.2.1SYMPTOMS: After creating an encryption zone and attempting to move data in this zone "Authentication Required" errors are reported in the kms-audit.log.2016-11-15 09:06:40,561 UNAUTHENTICATED RemoteHost:W.X.Y.Z Method:OPTIONS URL:http://hdp02.example.com:9292/kms/v1/keyversion/e1dw_dev_enc_key%400/_eek?eek_op=decrypt&doAs=test ErrorMsg:'Authentication required'The issue reproduces only in ambari-view and not via HDFS commands. Following errors are reported from browser.500 org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) ROOT CAUSE: For webhdfs to work with TDE, Ranger KMS must be configured to allow hdfs user to access all keys. This is a configuration issue in Ranger KMS. Ambari deploys default configuration where hdfs user is not allowed. This is a known behavior reported in BUG-45012<property> <name>hadoop.km