Community Articles

PROBLEM: Enable GC logging for Zookeeper SOLUTION: When Using Ambari web UI: 1. Click on the Zookeeper Service 2. Click on Configs tab 3. Navigate to 'Advanced Zookeeper-env'4. Locate the setting 'zookeeper-env template' 5. Append the following to 'export SERVER_JVMFLAGS=-Xmx1024m' : -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$ZOO_LOG_DIR/zookeeper.gc.`date +'%Y%m%d%H%M'`" To be precise it should look like :export SERVER_JVMFLAGS="-Xmx1024m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$ZOOKEEPER_LOG_DIR/zookeeper.gc.`date +'%Y%m%d%H%M'`" 6. Save the changes and restart the Zookeeper service when prompted When the cluster is managed outside Ambari: 1. On application cluster node open zookeeper-env.sh 2. Append the above mentioned parameter to SERVER_JVMFLAGS values. You can find the zookeeper-env.sh at /etc/zookeeper/conf/

SYMPTOMS: When using the SelectHiveQL processor in Nifi to run hive queries, it does not use the specific yarn queue ('nifi') if it is configured as in the following HiveConnectionPool settings:jdbc:hive2://<server>:<port>;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=kerberos;principal=hive/_HOST@DOMAIN.COM;tez.queue.name=nifiFollowing is an example of queue setting in yarn capacity scheduler:yarn.scheduler.capacity.root.queues=default,nifiyarn.scheduler.capacity.root.default.user-limit-factor=1yarn.scheduler.capacity.root.default.state=RUNNINGyarn.scheduler.capacity.root.default.maximum-capacity=40yarn.scheduler.capacity.root.default.capacity=40yarn.scheduler.capacity.root.default.acl_submit_applications=yarnyarn.scheduler.capacity.root.capacity=100yarn.scheduler.capacity.root.acl_administer_queue=yarnyarn.scheduler.capacity.root.accessible-node-labels=*yarn.scheduler.capacity.node-locality-delay=40yarn.scheduler.capacity.maximum-applications=10000yar

From HDP-2.5 onwards Ranger installation with Ambari 2.4.0, supports uploading of solrconfig to zookeeper during start of Ranger Admin component. However once default solrconfig.xml is uploaded the modified solrconfig.xml is not uploaded again for the next Ranger Admin start.To upload modified solrconfig.xml, below manual steps needs to be followed.1) Edit solrconfig.xml: Ranger solrconfig.xml is located at /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml2) Upload modified solrconfig.xml under znode used by Ranger Audit: Znode can be found in property ranger.audit.solr.zookeepers as {zookeeper_host}:2181/znode under Ranger Audit Tab.NOTE: If Ranger is using Ambari Infra solr cloud then default znode is /infra-solr otherwise if it is using externally managed solr cloud znode is /ranger_audits Use below command to upload, need to also export JAVA_HOME=/usr/jdk64/jdk1.8.0_77/usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string {zookee

PROBLEM : Cloudbreak failed to start. SYMPTOM : We get the below error "bind already in use"ERROR :ERROR: for consul Cannot start container 8a0b07e6b35e7350656b54314f7023107f417c0e0031db28e91f6bfb7a245d6f: failed to create endpoint cbreak_consul_1 on network bridge: Error starting userland proxy: listen udp X.X.X.X: bind: address already in use Traceback (most recent call last): File "<string>", line 3, in <module> File "compose/cli/main.py", line 63, in main AttributeError: 'ProjectError' object has no attribute 'msg' docker-compose returned -1 ROOT CAUSE : Cloudbreak may have stale process and because there are some container still running from the previous run. RESOLUTION : Please run the following : cbd kill # check if all containers have stopped cbd start This should fix the issue.

screen-shot-2017-03-09-at-44439-pm.pngscreen-shot-2017-03-09-at-44200-pm.pngPlease use the REST-plugin tool to perform the deletion. Please find the attached screenshot on how to use it, and the instruction on how to use encoded password (string to the word Basic) REST-plugin : https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo/related?hl=en-USYou can encode your password using this tool : http://base64-encoding.online-domain-tools.com/ Type in the format username:password to get the encoded password.

Why-Hive-rolling-restart-doesn-t-change-the-configurations-all-the-time?This completely depends on what configuration are you trying to change.For example : We changed the value of hive.exec.compress.output to true and restarted 1 hiveserver2 and when we checked beeline set hive.exec.compress.output the value got changed. 2. Then we changed the log4j property hive.root.logger value to Debug and then just restarted hiveserver2 but the change didn't take effect until we restart Hive metastore on the host 3. Then we changed hadoop.security.credential.provider.path this value but this didn't take effect until we refresh the client configs first and then restart metastore and Hiveserver2. (restart order is important)

STEPS :Take a backup of your Ambari database before executing the below command as you are issuing a delete command.curl --insecure -u admin:<password> -H 'X-Requested-By: ambari' -X DELETE http://<ambari-server-fqdn>:8080/api/v1/groups/<group name you wanted to delete>

An OutputCommitter that commits files specified in job output directory i.e. ${mapreduce.output.fileoutputformat.outputdir}. in mapred-site.xml The file output committer algorithm version valid algorithm version number: 1 or 2 default to 1 The file output committer has three phases 1.Commit task Recover task Commit Job If you choose version value to be Version 1 : 1. Commit task will rename the directory from $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/ to $joboutput/_temporary/$appAttemptID/$taskID/ , to put it in a simple way it just reduces one subdirectory. 2. RecoverTask will also do a rename of $joboutput/_temporary/$appAttemptID/$taskID/ to $joboutput/_temporary/($appAttemptID + 1)/$taskID/ 3. Commit Job will merge every task output in $joboutput/_temporary/$appAttemptID/$taskID/ to the path which is specified in the mapreduce.output.fileoutputformat.outputdir $joboutput/, then it will delete $joboutput/_temporary/ and write $joboutput/_SUCCESS It has a perfor

HDF Version: 2.1.2Test Environment: Single AWS VPC with multiple subnets, EC2 Security Groups with port rules controlling access between subnets, AWS LogFlow to track accepted/rejected traffic, 1 EC2 instance for Management, 3 EC2 instances for worker nodes, Laptop on public IP for remote access.Deployment / Testing Method: Installed baseline of Ambari, Ambari Infra, Ambari Metrics, NiFi Certificate Authority, Zookeeper Cluster, NiFi Cluster, ClientsAdded RangerEnabled SSL across all servicesTested all commonly used interfaces, checked for rejected trafficNot Tested: Using External LDAP/AD servicesUsing external Certificate AuthorityConnecting to Auxiliary servicesDeployment TopologyZoneM(anagement)D(ata)R(emote)P(ublic)MembersManagement NodeWorker Nodes 1,2,3Other NiFi ClusterUsersServicesAmbari Infra (Infra Solr), Ranger, Metrics Collector, Grafana, Metrics Monitor, Nifi Certificate Authority, Clients (Infra Solr, ZooKeeper)NiFi, Zookeeper, Metrics Monitor, Clients (Zookeper)NiFiBro

STATEMENT: Ambari audit log file gets generated from Ambari ver. 2.4 - https://issues.apache.org/jira/browse/AMBARI-15241. PROBLEM: How can I enable ambari audit in Ambari ver. 2.2.2? SOLUTION: To get ambari 2.2.2 having the audit, follow up the below instruction. This would be to change log level to DEBUG for this PersistKeyValueService class. - log into ambari node- edit /etc/ambari-server/conf/log4j.properties- set (or add) the following property:log4j.logger.org.apache.ambari.server.api.services.PersistKeyValueService=DEBUG- save the changes- run: $ ambari-server restart

It is known that in a secure Kafka environment, we need to explicitly authorize user/principal to either read/write to a Kafka topic as shown belowGrant Read/Write Access to a TopicTo add the following ACL:"Principals user:bob and user:alice are allowed to perform Operation Read and Write on Topic Test-Topic from Host1 and Host2"run the CLI with the following options:bin/kafka-acls.sh --add --allow-principal <strong><em>User</em></strong>:bob --allow-principal User:alice --allow-host host1 --allow-host host2 --operation Read --operation Write --topic test-topicGrant Full Access to Topic, Cluster, and Consumer GroupTo add ACLs to a topic, specify --topic <topic-name> as the resource option. Similarly, to add ACLs to cluster, specify --cluster; to add ACLs to a consumer group, specify --consumer-group <group-name>.The following examples grant full access for principal bob to topic test-topic and consumer group 10, across the cluster. Substitute your ow

A often requested feature has popped up in HDP 2.5, Phoenix namespace mapping AKA Phoenix schemas. This feature allows creation of schemas and development within a schema work space. This guide will walk through steps to enable this feature and how to use it.Use Case:Create 2 schemea named "schema1" and "scheama2"Within both schemas create table customer.Create another schema named "schema3". Drop schema "schema3"Note - A schema can only be dropped if it is empty (does not host any tables).1. Through Ambari go to HBase. Select Advanced tab 2. Add a parameter to hbase-site.xml. To do this go to "Custom hbase-site" and click on "Add Property"3. Add the property Key: phoenix.schema.isNamespaceMappingEnabled Value: trueClick on Add button. A restart hbase service will be required. 4. Test the functionality, use sqlline.py. It is located under /usr/hdp/current/phoenix/binNote - I am using /usr/hdp/2.5.0.0-1245/phoenix/bin since I want to prove to the audience I am in fact using HDP 2.

When Ranger service is configured for using custom PID file location and custom user using HDP-2.5, below steps need to be followed.1) Edit ranger-admin-services.sh script file: This file is located at /usr/hdp/current/ranger-admin/ews/ranger-admin-services.sh which have hard-coded default path as pidf=/var/run/ranger/rangeradmin.pid. Need to update the hard-coded path with the custom value specified from Ambari in ranger_pid_dir property under Advanced ranger-env section. Also update the hard-coded user used for chown {pidf} with the service user’s name given during installation of Ranger service on a Misc Tab from Ambari.Update PID location:pidf={custom_pid_path}/rangeradmin.pidUpdate service user:chown {custom_user} ${pidf}2) Edit ranger-usersync-services.sh: This file is located at /usr/hdp/current/ranger-usersync/ranger-usersync-services.sh. Similar changes are required to update the hard-coded pid location and hard-coded service user for Ranger Usersync component.Update PID locat

Here is the steps to access HDFS UI through Knox. This is tried and tested with Ambari 2.4 and HDP2.51. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.5.3.0-37/services3. download the configurations from https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/hdfsui/2.7.0/ URL.4. make sure your folder structure should look alike /var/lib/knox/data-2.5.3.0-37/services/hdfsui/2.7.0 and should have rewrite.xml and service.xml files5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.5.3.0-37/services/hdfsui/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service> <role>HDFSUI</role> <url>http://NNHOST:50070</url> </service>7.Restart Knox service.8. You should be able to access Name node UI from the below URL https://Knxo-host:8443/gateway/default

In the past, if Select returned no rows, hadoop would still create an empty file. Some customer would use this file to help with their work flow. In https://issues.apache.org/jira/browse/HIVE-13040 this was fixed, because anyone using cloud based solutions could be charged for empty files, so it was determined that hadoop should not write the files.

SYMPTOM Concurrency issues hit during multi-threaded moveFile issued when processing queries such as "INSERT OVERWRITE TABLE ... SELECT .." The following pattern is displayed in stack trace:Loading data to table testdb.test_table from hdfs://xyz/ra_hadoop/.hive-staging_hive_2017-01-31_14-09-52_561_8101886747064006778-4/-ext-10000 ERROR : Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2883) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3140) at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1727) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:353) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745) at org.apache.h

OverviewAmbari's Kerberos Descriptor is JSON-formatted document used to help Ambari enable Kerberos for installed services. The descriptor contains the information needed to create the required principals and keytab files. It also declares configuration changes needed by the serviced so they a are configured for Kerberos properly. The Kerberos Descriptor is comprised of the compiled Kerberos descriptors found in the relevant service definitions with user-specified changes applied to it. This combination of data is known as the "Composite Kerberos Descriptor", where the separate parts are known as the "Stack-level Kerberos Descriptor" and the "User-specified Kerberos Descriptor". Each of these descriptors may be obtained from Ambari via is REST API:GET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/COMPOSITEGET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/STACKGET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/USERNOTE: Be sure to replace {CLUSTER_NAME} when the name o

How to set up a Ranger enabled cluster using Ambari-blueprint. In order to setup a cluster using Ambari blueprints we will need a minimum three node cluster.Lets say this cluster contains three hosts as below:simple-ranger-a.comsimple-ranger-b.comsimple-ranger-c.com Pre-requisites: To setup a Ranger enabled cluster we need to install ambari-server on one of the hosts can be the called the gateway node let and ambari-agents should be installed on all the nodes.All the ambari-agents should be configured to point to the ambari-server on the gateway node.Note : The operating system used for this setup is Cent-OS.Lets consider the gateway node to be simple-ranger-a.com, to install ambari-server on this host we need to add ambari repository to the OS repositories. To update the OS repository we add ambari.repo file to /etc/yum.repos.d/ directory. Next we do a refresh of the OS repositories using command yum clean allThen we run the command yum install -y ambari-server ambari-agent, this wi

For users having hive insert query with dynamic partition, and partitions (> 10) on a column, you may notice that your query is generating too many small files per partition. INSERT OVERWRITE TABLE dB.Test partition(column5)select column1 ,column2 ,column3 ,column4 ,column5from Test2;For Example, if your table has 2000 partitions, and your query is generating 1009 reducers (hive.exec.reducers.max), then you might end up with 2 million small files. To Understand "How Does Tez determine the number of reducers" refer: https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html)This could result into issues with:1. HDFS Namenode performance:Refer: https://community.hortonworks.com/articles/15104/small-files-in-hadoop.html2. File Merge Operation failing due java.lang.OutOfMemoryError: GC overhead limit exceeded .» File Merge , java.lang.OutOfMemoryError: GC overhead limit exceeded   at java.lang.StringCoding$StringDecoder.decode(StringC

IoTWorking with IoT data is a many layered process, not unlike a parfait. Scratch that, an Onion. In fact, an Onion Omega2, which is a great device that I just got yesterday does IoT really easily. This is so much easier to setup that RPI or other platforms. It also has a ton of pluggable modules that stack on top of this small chip. It's pretty small powered, but it's under $10.The device is extremely well documented at their site.Needed to run real tools, I added a USB stick and used that for storage and for extra SWAP space.opkg updateopkg install kmod-usb-storage-extras e2fsprogs kmod-fs-ext4umount /tmp/mounts/USB-A1mkfs.ext4 /dev/sda1mkdir /mnt/sda1mount /dev/sda1 /mnt/sda1mount /dev/sda1 /mnt/ ; tar -C /overlay -cvf - . | tar -C /mnt/ -xf - ; umount /mnt/opkg updateopkg install block-mountopkg updateopkg install swap-utils block-mountdd if=/dev/zero of=/tmp/mounts/USB-A1/swap.page bs=1M count=256mkswap /tmp/mounts/USB-A1/swap.pageswapon /tmp/mounts/USB-A1/swap.pagefreeb

SYMPTOM: When you create an Oozie workflow which contains a SSH function, it can fail with an error of "Not able to perform operation [ssh -o PasswordAuthentication=no -o KbdInteractiveDevices=no -o StrictHostKeyChecking=no -o ConnectTimeout=20 root@localhost mkdir -p oozie-oozi/0000009-131023115150605-oozie-oozi-W/Ssh--ssh/ ] | ErrorStream: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)."   ROOT CAUSE: Passwordless SSH must be setup correctly from oozie@server1 to $user@server2For example:  RESOLUTION: If an Oozie workflow contains an ssh command from server 1 to server 2 as root, then the passwordless SSH must be setup as the following: oozie@server1 > root@server2   This article created by Hortonworks Support (Article: 000001722) on 2015-04-01 11:05 OS: Linux Type: Configuration,Executing_Jobs

Depends on whether we are talking about under HDFS or at the OS level, but the answer to both is Yes...For accounts under HDFS, the home directory is determined by the value of dfs.user.home.dir.prefix. see: https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xmlFor OS users you can pre-create your service users before install, then set ignore_groupsusers_create=true in cluster-env.xml, and then launch the installer. The service users and groups are here: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_command-line-installation/content/create-system-users-and-groups.html

This tutorial will demonstrate how to execute PySpark jobs on an HDP cluster and pass in parameter values using the Livy REST interface. I will also demonstrate how to interact with Livy via Apache Zeppelin and use forms in Zeppelin to pass in parameter values. Livy is an open source REST interface for interacting with Spark. Livy requires at least Spark 1.6. By default, Livy is built against Apache Spark 1.6.2 with HDP 2.5.3 If on HDP 2.4.2, then Livy 0.2.0 might be part of HDP 2.4.2 repos. If not, then it can be cloned from: https://github.com/cloudera/livy Livy is built against Apache Spark 1.6.2 with HDP 2.5.3. The following was tested with HDP 2.5.3 and Spark 1.6.2. Business Use Case: A PySpark script that resides on the Livy server aggregates order statistics from a Hive table for a date range passed in as parameters to the Python script. The Start and End dates are passed in as parameter values to the Python script. The job runs on the cluster via YARN and produces the followi

it is not completely obvious but you can certainly run Python scripts within Oozie workflows using the Shell action.Here's a sample job.properties file, nothing special about it.nameNode=hdfs://sandbox.hortonworks.com:8020jobTracker=sandbox.hortonworks.com:8050queueName=defaultexamplesRoot=oozieoozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/pythonHere's a sample workflow that will look for a script called script.py inside scripts folder<workflow-app xmlns="uri:oozie:workflow:0.4" name="python-wf"> <start to="python-node"/> <action name="python-node"> <shell xmlns="uri:oozie:shell-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </p

When upgrading to HDP 2.5 or later with Falcon, you might encounter the following error when starting the ActiveMQ server:
ERROR - [main:] ~ Failed to start ActiveMQ JMS Message Broker. Reason: java.lang.NegativeArraySizeException (BrokerService:528)You might also encounter this error if downgrading Falcon from HDP 2.5 to an earlier release. If you encounter this error, delete the ActiveMQ history and then restart Falcon. If you want to retain the history, be sure to back up the ActiveMQ history prior to deleting it.
 cd <ACTIVEMQ_DATA_DIR>rm -rf ./localhostcd /usr/hdp/current/falcon-server
su -l <FALCON_USER>


./bin/falcon-stop
./bin/falcon-start