Community Articles

SYMPTOM: When trying to enable the Ranger plugin for the HIVE component and then clicking SAVE, it does not save this as a new configuration.Certain Hive Smart Config changes cannot be made (set Authorization to Ranger, etc) when Oracle JDBC URL contains a non-standard port. ROOT CAUSE: https://hortonworks.jira.com/browse/BUG-50133 WORKAROUND: Reach out to Hortonworks Support for a hotfix patch and instructions to address this issue.

SYMPTOM: Nagios install fails due to existing "nagios" user in the ldap systemtrying to install ambari on single node cluster but is having issues with nagios since they have an ldap server with a nagios user already created.ERROR: Receiving error; "err: /Stage2/Hdp-nagios::Server/Hdp::Usernagios/Usernagios/gid: change from 20000 to nagios failed: Could not set gid on usernagios: Execution of '/usr/sbin/usermod -g 492 nagios' returned 6: usermod: user 'nagios' does not exist in /etc/passwd ROOT CAUSE: This is BUG - https://hortonworks.jira.com/browse/BUG-6787 RESOLUTION / WORKAROUND:Workaround #1: That user must be in group = nagios as well, or the nagios install fails. The workaround for this is to add nagios user to a nagios group.Workaround #2: During install, customize nagios user (Customize Services > Misc) and use something other than nagios user. This user will be created and put in a group nagios. This is available in HDP 1.3.1

Problem Description: often we are in need to read and write underlying files from user defined reader and writer. if the the custom reader and writer are written in java or language run over JVM then we are good to add those in hive_aux_jar or we can add them using add jar option at session level but when if it is written in native language and shipped as *.so file then we will get java.lang.UnsatisfiedLinkError. we can workaround this problem after adding it to hive-env 1. open Ambari-->Hive-->Advanced-->Advanced hive-env-->hive-env template 2. modify {% if sqla_db_used or lib_dir_available %} export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:{{jdbc_libs_dir}}" export JAVA_LIBRARY_PATH="$JAVA_LIBRARY_PATH:{{jdbc_libs_dir}}" {% endif %}

SYMPTOM: External group synced from Unix/LDAP/AD system are labelled as ‘Internal’ in Ranger Admin UI.Currently, when I check under Ranger -> Settings -> Groups, I see around 1700 groups. And all the groups are shown as internal. However, when I login to the Linux hosts I see only 77 groups. I am wondering how to identify from UI which groups are internal and external as all the groups are shown as internal. ERROR: 1. After Ranger and Usersync installation, started Ranger admin and Usersync to sync UNIX OS users. 2. Logged in to Ranger Admin Dashboard and visited User/Group section, then clicked on Group tab.External group synced from Unix/LDAP/AD system are labelled as ‘Internal’ in Ranger Admin UI. ROOT CAUSE: This bug is targeted for inclusion in the Release Notes>Known Issue in HDP 2.3.4, as it has fixVersion in (Dal-M21, Dal-M30, DAL-M30, Dal-Next)https://hortonworks.jira.com/browse/BUG-47662 [HWX internal url for tracking purpose] https://reviews.apache.org/r/42146/ RE

Question: We would like to Understand the mechanism behind the properties yarn.scheduler.capacity.node-locality-delay==> All the YARN schedulers try to honor locality requests. On a busy cluster, if an application requests a particular node, there is a good chance that other containers are running on it at the time of the request. The obvious course of action is to immediately loosen the locality requirement and allocate a container on the same rack. However, it has been observed in practice that waiting a short time (no more than a few seconds) can dramatically increase the chances of being allocated a container on the requested node, and therefore increase the efficiency of the cluster. This feature is called delay scheduling, and it is supported by both the Capacity Scheduler and the Fair Scheduler. Every node manager in a YARN cluster periodically sends a heartbeat request to the resource manager—by default, one per second. Heartbeats carry information about the node manager’s

SYMPTOM: While following steps for HDP pre-upgrade activity the active namenode went down while issuing - #hdfs dfsadmin -savenamespace" command.Below was the error -================================================ hdfs@namenode1~$ hdfs dfsadmin -saveNamespace saveNamespace: Call From namenode1.example.com/10.160.81.30 to namenode1.example.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused================================================ERROR: 2016-06-14 02:18:49,774 WARN ha.HealthMonitor (HealthMonitor.java:doHealthChecks(209)) - Transport-level exception trying to monitor health of NameNode at namenode1.example.com/10.10.20.30:8020: Call From namenode1.example.com/10.10.20.30 to namenode1.example.com:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused2016-06-14 02:18:51,774 INF

SYMPTOM : While installing Ranger through Ambari , on Oracle DB, user runs into error "sqlplus64: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directory" ERROR:sqlplus64: error while loading shared libraries: libsqlplus.so: cannot open shared object file: No such file or directoryROOT CAUSE : possible missing environment settings and misconfiguration in Ambari .RESOLUTION : 1. On Ambari Server Machine, please check environment variable in Linux OS level, verify environmental path LD_LIBRARY_PATH=/usr/lib/oracle/<version>/client64/lib is set. 2. AND in Ambari config -> ranger config - advanced ranger-env, set property "oracle_home" with "/usr/lib/oracle/<version>/client64/lib"

Ambari Server Performance - This alert appears when the size of the below API call does not respond back within the threshold limit configured in the alert notifications. The reason for this could be the content of the API response just got bigger over the releases. curl: http://<host>:<port>api/v1/clusters/<clustername>; Ambari alert sample: Alert 1: (Time : 01 December 2016 19:42) CRITICAL Ambari Server Performance Performance Overview: Database Access (Request By Status): 1ms (OK) Database Access (Task Status Aggregation): 1ms (OK) REST API (Cluster): 16,454ms (CRITICAL) Cluster: <clustername> Alert2 : (Time : 02 December 2016 10:40 ) Ambari Server Performance Performance Overview: Database Access (Request By Status): 1ms (OK) Database Access (Task Status Aggregation): 1ms (OK) REST API (Cluster): 8,544ms (CRITICAL) Cluster: <clustername> Possible fix: Increase the threshold limit configured in the alert notification or disable the alert. In future r

SYMPTOM: During an upgrade to Ambari 2.1.2 and HDP 2.3.2 the upgrade fails when trying to start the namenode service. Errors in the stack trace indicate that the xasecure files are not found.ISSUE: When Ranger has been installed outside of Ambari, the method used for previous HDP versions, the Ambari database does not contain information about Ranger. As a result the configuration files are not replicated to the new directory structure, /etc/hadoop/<version>/0/. Since the Ranger properties are included in other component configuration files the namenode service will not start. A result of the missing configuration files for Ranger.RESOLUTION: If Ranger was installed outside of Ambari you can replicate the xasecure files from the previous version to the current version configuration directory. cp /etc/hadoop/conf/xasecure* /etc/hadoop/<version>/0/ ie. cp /etc/hadoop/conf/xasecure* /etc/hadoop/2.3.2.0-2950/0/ ** You will need to complete this on all namenode servers in t

SYMPTOMS: Although valid Kerberos ticket is available, we are unable to put files in HDFS encrypted zone. If we restart/failover namenode, then we are able to put files with the same ticket/credentials without having to get a new ticket. Below is the demo of the issue where /tmp/user1 is the encrypted zone and user has permission to that zone: [root@test ~]# su - user Last login: Thu Oct 13 13:03:24 EDT 2016 on pts/57 -bash-4.2$ id uid=11516(user) gid=5000(bns) groups=5000(bns),1520(cmtsuser),1800(admin),4534(edgegrp),4535(edgedgrp),4536(k2tstgrp),8242(ftallocctxd),8243(ftallocctxu),15113(hdpadm) -bash-4.2$ kinit Password for user@123.EXAMPLE.COM: -bash-4.2$ klist Ticket cache: FILE:/tmp/krb5cc_11516 Default principal: user@123.EXAMPLE.COMValid starting Expires Service principal 10/14/2016 07:23:51 10/14/2016 17:23:51 krbtgt/123.EXAMPLE.COM@EXAMPLE.COM renew until 10/21/2016 07:23:48 -bash-4.2$ hadoop fs -put file1 /tmp/user1/file_1 put: java.util.concurrent.ExecutionException: java.io

SYMPTOMS: When local disk utilization of multiple node managers goes high beyond a limit, nodes turn “unhealthy” and gets into the "blacklist" not to be used for container/task allocation, hence reducing the effective cluster capacity.ROOT CAUSE: A burst or rapid rate of submitted jobs with substantial NM usercache resource localization footprint may lead to rapid fill up of the NM local temporary file system with negative consequences in terms of stability. The core issue seems to be the fact that NM continues to localize the resources beyond the maximum local cache size (yarn.nodemanager.localizer.cache.target-size-mb , default 10G). Since maximum local cache size is effectively not taken into account when localizing new resources (note that default cache cleanup interval is 10 min controlled by yarn.nodemanager.localizer.cache.cleanup.interval-ms), this basically leads to sort of self-destruction scenario : Once the filesystem utilization reaches the threshold of 90%, NM will autom

SYMPTOM: We had problem with namenode starts. We can check if there is namenode startup issue then we can check for gc related errors in logs and same time check for list of files/blocks customer is having on cluster and the recommended HEAP size as per hortonworks manual - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html ROOT CAUSE: GC errors were causing the issue for namenode to startRESOLUTION: Below are the steps taken to resolve the issue -1. Logged in to the namenode cli2. When checked from cli using “ps –ref |grep –i namenode”, the namenode was not displaying. 3. Seems that the namenode process was getting killed after specific interval of time, but ambari was still showing the namenode process state in UI as “starting” 4. Cancelled the namenode starting process from Ambari UI. 5. Tried starting the whole HDFS process and simultaneously ran “IOSTAT” on the fsimage disk. 6, We found withi

Problem Statement: when doing an ambari-server sync-ldap -groups=<your file>It will bring over the groups but not the users in it. ROOT CAUSE: When troubleshooting why the group members are not being sync'd with FreeIPA, a packet trace helped identify the issue. With ActiveDirectory the user's DN is exposed as an attribute: "distinguishedName", this is not the case inFreeIPA/RHEL IDM (using 389 DS for the directory server implementation). The DN is not an attribute on the user, and cannot be used in a filter like this:(&(objectClass=posixaccount)(|(dn=uid=dstreev,cn=users,cn=accounts,dc=hdp,dc=local)(uid=uid=dstreev,cn=users,cn=accounts,dc=hdp,dc=local)))If we want to retrieve a specific object by DN we have to set the DN as the search base and do a base search scope.ldapsearch -H ldap://ad.hortonworks.local:389 -x -D "CN=hadoopsvc,CN=Users,dc=hortonworks,dc=local" -W -b "CN=paul,CN=Users,DC=hortonworks,DC=local" -s base -a always "(objectClass=user)"In this case I'm looking

SYMPTOM: oozie-hive job is running very slow, sometimes jobs are stuck is in final stage and was not able to completeROOT CAUSE:Oozie prepares the hive-site.xml for hive action, it has the mapred parameter mapreduce.job.reduces set to 1 by default,the reason for this is oozie prepare action configuration after reading core-site.xml,hdfs-site.xml,mapred-site.xml etc.with the setting of mapreduce.job.reduces=1 the job is running with single reducer hence taking a lot of time to complete.WORKAROUND:set mapreduce.job.reduces to -1RESOLUTION:there is oozie fix https://issues.apache.org/jira/browse/OOZIE-2205 enhance actionConf which are passed to hive-action.

SYMPTOM: When Ambari is upgraded to 2.4.x, SmartSense has to be upgraded before upgrading the HDP stack.Failing to do so, would cause SmartSense to capture bundle with small size (in KB's) that has very less data.ERROR: ERROR 2016-12-01 23:02:02,279 anonymize.py:63 - Execution of command returned 1. Exception in thread "main" java.lang.NullPointerException at com.hortonworks.smartsense.anonymization.rules.RuleType.from(RuleType.java:47) at com.hortonworks.smartsense.anonymization.rules.Rules.createRule(Rules.java:161) at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:148) at com.hortonworks.smartsense.anonymization.rules.Rules.createRules(Rules.java:132) at com.hortonworks.smartsense.anonymization.AnonymizerFactory.createAnonymizer(AnonymizerFactory.java:94) at com.hortonworks.smartsense.anonymization.Main.start(Main.java:202) at com.hortonworks.smartsense.anonymization.Main.main(Main.java:294)ERROR 2016-12-01 23:02:02,280 AnonymizeBundleCommand.py:60 - Ano

ISSUE : If SOLR is configured as backend storage for ranger audits, ranger audit screen shows incorrect screen. This happens if server is running on any timezone other than UTC.ROOT CAUSE :'ranger-admin-services.sh' script uses below command to start ranger.nohup java -Dproc_rangeradmin ${JAVA_OPTS} -Dlogdir=${RANGER_ADMIN_LOG_DIR} -Dcatalina.base=${XAPOLICYMGR_EWS_DIR} -cp "${XAPOLICYMGR_EWS_DIR}/webapp/WEB-INF/classes/conf:${XAPOLICYMGR_EWS_DIR}/lib/*:${RANGER_JAAS_LIB_DIR}/*:${RANGER_JAAS_CONF_DIR}:${JAVA_HOME}/lib/*:${RANGER_HADOOP_CONF_DIR}/*:$CLASSPATH" org.apache.ranger.server.tomcat.EmbeddedServer > ${RANGER_ADMIN_LOG_DIR}/catalina.out 2>&1 &If JAVA_OPTS value does not contain '-Duser.timezone=UTC', then Ranger admin runs in the local timezone. This causes timezone issue in viewing audit on Ranger UI as Ranger admin doesn't return eventtime in UTC format.RESOLUTION : This is fixed under https://issues.apache.org/jira/browse/RANGER-1249WORKAROUND :1. Create a file

SYMPTOM: We are connected to hue using hdfs user. After executing oozie w/f using HUE the workflow is failing.Below is sample job definition -<workflow-app name="Headlight" xmlns="uri:oozie:workflow:0.4"> <start to="Headlight"/> <action name="Headlight"> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <exec>/user/hdfs/oozie/deployments/headlight-exec-all.pl</exec> </shell> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> ERROR: Tried creating simple script and ran, but the files are created using yarn user as shown below -#!/bin/sh /bin/touch /tmp/test.oozie The result is ok. the file is created by the user yarn. -rw-r--r-- 1 yarn hadoop 0 Jun 3 03:04 /tmp/

ENVIRONMENT: HDP 2.3.4, Ambari 2.2.1SYMPTOMS: After creating an encryption zone and attempting to move data in this zone "Authentication Required" errors are reported in the kms-audit.log.2016-11-15 09:06:40,561 UNAUTHENTICATED RemoteHost:W.X.Y.Z Method:OPTIONS URL:http://hdp02.example.com:9292/kms/v1/keyversion/e1dw_dev_enc_key%400/_eek?eek_op=decrypt&doAs=test ErrorMsg:'Authentication required'The issue reproduces only in ambari-view and not via HDFS commands. Following errors are reported from browser.500 org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) ROOT CAUSE: For webhdfs to work with TDE, Ranger KMS must be configured to allow hdfs user to access all keys. This is a configuration issue in Ranger KMS. Ambari deploys default configuration where hdfs user is not allowed. This is a known behavior reported in BUG-45012<property> <name>hadoop.km

SYMPTOM:hive CLI is in hung state for so long, an impatient user did CRTL+C to exit of it and complain about the hive CLI slowness.ROOT CAUSE: user is running hive CLI in Kerberos enabled security, we asked them to enable debug logging at the console using hive --hiveconf hive.root.logger=debug,console and saw the following GSS exception due to ticket expiration.WARN hive.metastore: Failed to connect to the MetaStore Server...org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Na

ROOT CAUSE: YARN UI shows total memory vs used memory wrong when there are reserved resources. The memory total shown when there is no reserved resources will be the correct one. We can also compare that with sum of all NodeManager memory resource. This won't have any impact on YARN scheduler logic. This behavior has been reported in bug in YARN UI and got fixed by https://issues.apache.org/jira/browse/YARN-3432 and https://issues.apache.org/jira/browse/YARN-3243 REFERENCES: https://issues.apache.org/jira/browse/YARN-3432https://issues.apache.org/jira/browse/YARN-3243

It would sometimes be useful to adjust the log level when executing hbase client commands from command line without having to restart components. This article explains how that can be achieved:You can set DEBUG level for any client hbase command by exporting the following environment variable before it is executed:HBASE_ROOT_LOGGER=hbase.root.logger=DEBUG,consoleFor example:# export HBASE_ROOT_LOGGER=hbase.root.logger=DEBUG,console# hbase hbck2016-12-24 15:18:03,240 DEBUG [main] util.Shell: setsid exited with exit code 02016-12-24 15:18:03,363 DEBUG [main] lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, about=, sampleName=Ops, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)2016-12-24 15:18:03,373 DEBUG [main] lib.MutableMetricsFactory: field org.apache.hadoop.m

It would sometimes be useful to adjust the log level when executing hive commands from command line without having to restart components. This article explains how that can be achieved:You can provide the log level on the command line to hive as followshive -hiveconf hive.root.logger=DEBUG,console -e "show tables ;"For example:# hive -hiveconf hive.root.logger=DEBUG,console -e "show tables ;" |more16/12/24 15:10:23 DEBUG util.VersionInfo: version: 2.7.3.2.5.3.0-3716/12/24 15:10:27 [main]: DEBUG common.LogUtils: Using hive-site.xml found on CLASSPATH at /etc/hive/2.5.3.0-37/0/hive-site.xml16/12/24 15:10:27 [main]: DEBUG session.SessionState: SessionState user: nullLogging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties16/12/24 15:10:27 [main]: INFO SessionState: Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties16/12/24 15:10:27 [main]: DEBUG parse.VariableSubstitution: Substitution is on: hive16/12/24 15:10:

It's sometimes useful to adjust the log level when executing commands from the client to get more info about what is happening. This article explains how that can be achieved: You can set DEBUG level for any client command by exporting the following environment variable before it is executed: HADOOP_ROOT_LOGGER=hadoop.root.logger=DEBUG,console Now try executing a client command and watch the stream of DEBUG info come to your terminal. For example: # hdfs dfs -ls16/12/24 15:01:34 DEBUG util.Shell: setsid exited with exit code 016/12/24 15:01:34 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/2.5.3.0-37/hadoop/hadoop-common-2.7.3.2.5.3.0-37.jar!/core-default.xml16/12/24 15:01:34 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@149494d816/12/24 15:01:34 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/2.5.3.0-37/0/core-site.xml16/12/24 15:01:34 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream

SYMPTOMS: /tmp filling up causes multiple services to stop functioning. ROOT CAUSE: The issue happens due to internal Smartsense bug ST-2551.SOLUTION: Upgrade to Smartsense 1.3.1 WORKAROUND: To workaround this issue we need to manually modify two files related to Smartsense, so that the tmp files will not be generated in /tmp directory anymore 1. File : /usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py Change from : ANONYMIZATION_JAVA_COMMAND = "{0}" + os.sep + "bin" + os.sep + "java" +\ " -Dlog.file.name="+ ANONYMIZATION_LOG_FILE_NAME +\ " -cp {1} {2} {3}" Change to : ANONYMIZATION_JAVA_COMMAND = "{0}" + os.sep + "bin" + os.sep + "java" +\ " -Djava.io.tmpdir=/grid/02/smartsense/hst-agent/data/tmp" +\ " -Dlog.file.name="+ ANONYMIZATION_LOG_FILE_NAME +\ " -cp {1} {2} {3}" Make sure the tmp dir value is same as this property agent.tmp_dir in hst-agent-conf.2. File : /usr/sbin/hst-server.py Change from : SERVER_START_CMD = "{0}" + os.sep + "bin" + os.sep +\ "java -server -XX:NewRat

ENVIRONMENT: All Ambari versions prior to 2.4.xSYMPTOMS: Intermittent loss of heartbeat to cluster nodes, freeze of ambari-agent service, intermittent issues in Ambari alerts and service status updates in Ambari dashboard.Ambari-agent logs:- INFO 2016-08-21 19:10:20,080 Heartbeat.py:78 - Building Heartbeat: {responseId = 139566, timestamp = 1471821020080, commandsInProgress = False, componentsMapped = True}ERROR 2016-08-21 19:10:20,102 HostInfo.py:228 - Checking java processes failedTraceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/HostInfo.py", line 211, in javaProcs cmd = open(os.path.join('/proc', pid, 'cmdline'), 'rb').read()IOError: [Errno 2] No such file or directory: '/proc/24270/cmdline'Top command output: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP TIME DATA COMMAND 10098 root 20 0 54.4g 53g 4540 S 54.5 14.0 18000:11 224 300,00 54g /usr/bin/python2 /usr/lib/python2.6/site-packages/ambari_agent/main.py start --expected-hostname=12