Community Articles

A quick easy way to export users from LDAP to a CSV for use in importing to Ambari.ldapsearch -h ldapserver.mycompany.com -p 389 -D bind-user -w bind-password -b OU=London\ Users,OU=my\ dept,DC=mycompany,DC=com -s sub "(objectClass=user)" | grep sAMAccountName | sed 's/sAMAccountName://g' | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'

NOTE: This is for information and testing purposes only. Pleaes exercise caution prior to using it on business critical environments.STEPS:To completely remove HDP stack from the Operating System, do the following:1. Stop all the services in Ambari UI.2. Shutdown ambari-server and ambari-agent on Ambari server host using the following commands:$ ambari-server stop$ ambari-agent stop3. Shutdown ambari-agent on all other nodes using the following commands:$ ambari-agent stop4. Run the following command to check the installed services to delete the related configs and directories:$ hdp-select | grep -v 'None'5. Follow this URLto uninstall each services that is installed as per the above step or from the Ambari UI.6. Remove all the files and directory related with Hadoop and HDP using the following commands. This step will delete all the files and directories name with hadoop under root partition. Please be extra caution to run this step and make sure you have nothing important in these di

Note: This is for information and testing purposes only. Please exercise caution prior to using it on business critical environments.Steps to migrate the ranger and ranger_audit database to the new MySQL host without re-installing Ranger Service via Ambari UI - 1. Stop all the services on the cluster via Ambari UI.2. Make sure Ranger Admin and Ranger usersync is down.3. Take dump of ranger and ranger_audit databases.$ mysqldump ranger > /path/to/ranger_dump.sql$ mysqldump ranger_audit > /path/to/ranger_audit_dump.sql 4. Login to new database host and to the database.$ mysql -u root -p 5. Create user rangerdba and provide them all privileges.CREATE USER 'rangerdba'@'localhost' IDENTIFIED BY '<rangerdba_password>'; GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost'; CREATE USER 'rangerdba'@'%' IDENTIFIED BY '<rangerdba_password>'; GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%'; GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'<new_database_fqdn>';GRANT ALL PRIVILEG

In the current version of Kafka, when Kafka cluster is enabled with ranger authorizer(authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer) it is not possible to auto create topics as a non super user even if the auto create topic flag is set to true. In other words, Kafka create topic authorization can not be done at a topic level.For example, create a ranger policy as below, Topic AutoCreateTopic_Test* with all permissions to a non super user. Run the command line Kafka producer script to create a non existing topic, /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list < > --topic AutoCreateTopic_Test01--security-protocol PLAINTEXTSASL[2017-02-24 19:10:30,232] WARN Error while fetching metadata [{TopicMetadata for topic test4 -> No partition metadata for topic test4 due to kafka.common.TopicAuthorizationException}] for topic [test4]: class kafka.common.TopicAuthorizationException (kafka.producer.BrokerPartitionIn

PROBLEM : Hive view errors out with null pointer exception from Stand alone ambari view server. ERROR :java.lang.NullPointerExceptionat org.apache.ambari.view.hive2.resources.jobs.atsJobs.ATSParser.getHiveQueryIdFromJson(ATSParser.java:113)at org.apache.ambari.view.hive2.resources.jobs.atsJobs.ATSParser.getHiveQueryIdByOperationId(ATSParser.java:107)at org.apache.ambari.view.hive2.resources.jobs.Aggregator.readATSJob(Aggregator.java:260)at org.apache.ambari.view.hive2.resources.jobs.JobService.jsonObjectFromJob(JobService.java:161)at org.apache.ambari.view.hive2.resources.jobs.JobService.getOne(JobService.java:145)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ROOT CAUSE : Stand alone ambari host name missing in yarn.timeline-service.http-authentication.proxyuser.ambari-server. WORKAROUND / RESOLUTION : Update below property in YARN configs and make the value to *:yarn.timeline-service.http-authentication.proxyuser.ambari-server-<CLUSTER_NAME>.hosts=*yarn.timeline

Assuming you start with a kerberized HDP cluster with Hbase installed.First check what your hbase service principal is i.e.klist -kt /etc/security/keytabs/hbase.service.keytabKeytab name: FILE:hbase.service.keytabKVNO Timestamp Principal---- ----------------- -------------------------------------------------------- 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM 2 12/20/16 13:51:21 hbase/hdp253k1.hdp@HDP.COM In Ambari head to Hbase -> Configs -> Advanced -> Custom Hbase-Site.xml and add the following new parameters with the keytab / principal substituted for your environment. If they already exist for your cluster set the values as indicated:hbase.rest.authentication.type=kerberoshbase.master.kerberos.principal=hbase/_HOST@HDP.COMhbase.master.keytab.file=/etc/security/keytabs/hbase.service.keytabhadoop.proxyuser.HTTP.grou

Introduction The Hortonworks Sandbox running on Azure requires opening ports a bit differently than when the sandbox is running locally on Virtualbox or Docker. We’ll walk through how to open a port in Azure so that outside connections make their way into the sandbox, which is a Docker container inside an Azure virtual machine. Note: There are multiple ways to open ports to a Docker container (i.e. the sandbox). This tutorial will cover the simplest method, which reinitializes the sandbox to its original state. In other words, you will lose all changes made to the sandbox. Prerequisites Deploying Hortonworks Sandbox on Microsoft Azure Outline SSH Into The Azure VM Add Ports to the Docker Script Remove the Current Sandbox Container) Restart the Azure VM (Optional) Add New Ports to the SSH Config SSH Into the Azure VM If you followed the previous tutorial, Deploying Hortonworks Sandbox on Microsoft Azure, this step is as easy as running: ssh azureSandbox

PROBLEM : When the property skip.header.line.count is enabled on tables with large text files (~100G), we can see that the tez.grouping* parameters are ineffective, spinning one container per file.EXAMPLE: We have a table with 10 files, 117.0 G A.txt 29.0 G B.txt 30.0 G C.txt 1.9 G D.txt 11.0 G E.txt 18.9 G F.txt 5.7 G G.txt 159.4 H.txt 2.4 G I.txt 214.4 M J.txt Created table on top of this data, using CREATE TABLE `tab1`( `opr_glbl_sku_id` string, `opr_loc_nbr` string, `wk_start_dt` string, `base_fcst` float) ... TBLPROPERTIES ( 'skip.header.line.count'='1') Set these params (below), after which expectation is at least 200G/0.5G= 400 containers should be launched. SET tez.grouping.min-size=53421772; SET tez.grouping.max-size=534217728; With tez as the execution engine an attempt to perform select count(*) on the table spins only 10 containers. Once we remove the skip.header.line.count', it launched about 540 containers, which is expected behavior.ROOT CAUSE : hive.tez.i

1. Determine the version of the source hive database (from old cluster): mysql> select * from VERSION; +--------+----------------+----------------------------+ | VER_ID | SCHEMA_VERSION | VERSION_COMMENT | +--------+----------------+----------------------------+ | 1 | 1.2.0 | Hive release version 1.2.0 | +--------+----------------+----------------------------+ 2. On the old clusters metastore Mysql Database, take a dump of the hive database mysqldump -u root hive > /data/hive_20161122.sql 3. Do a find and replace in the dump file for any host name from the old cluster and change them to the new cluster (i.e. namenode address). Be cautious if the target cluster is HA enabled, when replacing hostname/hdfs url.4. On the new cluster, stop the Hive Metastore service 5. Within MySQL, perform drop/create of the new hive database. mysql> drop database hive; create database hive; 7. Run the schematool to initialize the schema to the exact version of the hive schema on the ol

PROBLEM : Query fails with the below mentioned exception.ERROR :java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:192)at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:131)ROOT CAUSE : Query fails on a field of datatype timestamp within an ORC table having null values, provided Predicate push down is enabled. This is a known issue. Reference: https://hortonworks.jira.com/browse/BUG-47472WORKAROUND / RESOLUTION : This setting is whether to enable automatic

In this article, we will see how to perform the mirroring of Hive data and metadata using Falcon from source cluster to destination cluster. This article is based on HDP 2.5. Configure HiveConfigure source and target Hive by clicking “Hive” from the Ambari Services menu, then click “configs” to add following custom properties on Ambari UI by scroll down to “Custom hive-site”, click it and then click “Add Property”.Add following property name with value:hive.metastore.event.listeners = org.apache.hive.hcatalog.listener.DbNotificationListenerhive.metastore.dml.events = truePress OK to save the changes, then click Restart all the impacted services. Bootstrap Table and DBBefore creating Hive DR mirroring job to replicate Hive data/metadata for a DB or Table, it is required to perform initial bootstrap of Table and DB from source to target cluster. Table BootstrapFor bootstrapping table replication, do an EXPORT of the table in question at source cluster, distcp the export directory t

There are many ways to validate a json file against a avro schema to verify all is kosher. Sharing a practice I have been using for few years.Objective - Validate avro schema well bound to the json fileFirst you must have a avro schema and json file. From there download the latest a avro-tools jar. At the moment 1.8.1 is the latest avro-tools version jar available.Store the avro schema and json file in the same directory. Issue a wget to fetch the avro-tools jarwget http://www.us.apache.org/dist/avro/avro-1.8.1/java/avro-tools-1.8.1.jarHere is what the directory looks likeObjective Details - Validate avro schema student.avsc binds to student.jsonHow - Issue the followingjava -jar ./avro-tools-1.8.1.jar fromjson --schema-file YourSchemaFile.avsc YourJsonFile.json > AnyNameForYourBinaryAvro.avroUsing the student files example:java -jar ./avro-tools-1.8.1.jar fromjson --schema-file student.avsc student.json > student.avroValidation passed, a avro binary was created.Now as a la

After defining cluster roles for the users and groups, LogSearch UI is not accessible with login failure message for individual users and groups.Below was the error in logs - logsearch-audit.json{"level":"WARN","file":"LogsearchAuthenticationProvider.java","thread_name":"qtp1464642111-13","line_number":124,"log_message":"{\"principal\":\"sysair\",\"result\":\"denied\",\"reason\":\"Wrong password\",\"remote_ip\":\"192.168.0.201\",\"session\":\"1a52ertsmv7l31qpo3gl3hdsyt\",\"auth_class\":\"org.springframework.security.authentication.UsernamePasswordAuthenticationToken\",\"user\":\"sysair\"}","logger_name":"org.apache.ambari.logsearch.audit","logtime":"1483536859578"} logsearch.json{"level":"INFO","file":"LogsearchAuthenticationProvider.java","thread_name":"qtp1464642111-14","line_number":66,"log_message":"Authenticating user:sysair, userDetail\u003dorg.springframework.security.authentication.UsernamePasswordAuthenticationToken@b48163fb: Principal: sysair; Credentials: [PROTECTED]; Authen

OverviewI recently encountered a question where someone asked to see how you can do preemption across YARN queues when a spark job is beyond it's queue's min guarantee. They had seen this before with the Fair Scheduler and Map Reduce, but wanted to apply the same experience here but with Spark and the Capacity Scheduler. This how-to article describes how to setup this experience. Goal: Run large Spark jobs in two separate capacity queues to produce an equal share of resources for both jobs. Hardware: 5 Nodes of AWS EC2 r3.xlarge Cluster Configuration: HDP: 2.4.2, Spark: 1.6.1, 5 Node Managers, 20GB (20480MB) Yarn Containersyarn.scheduler.maximum-allocation-mb = 20480yarn.scheduler.minimum-allocation-mb = 2560 High Level Setup:1. Add preemption properties as per documentation2. Create Two YARN Queues with Fair OrderingChild queue “test1” with a min capacity of 50% and a max of 100%Child queue “test2” with a min capacity of 50% and a max of 100%Root queue with a fair ordering policy3.

If you upgrade to or install HDP 2.5.0 or later without first installing the Berkeley DB file, you will get the error "Unable to initialize Falcon Client object. Cause : Could not authenticate, Authentication failed". The information below has also been added to the Data Movement and Integration guide and other Hortonworks documentation.Prerequisite to Installing or Upgrading Falcon Starting with the HDP 2.5.0 release, customers must provide the free Berkeley DB JAR file, under the Oracle license, as part of a Falcon installation or upgrade. This applies to Ambari express or rolling upgrades, to manual upgrades, and to all installations of HDP 2.5.0 or later, if you are using the Falcon service. Following are both Ambari and manual instructions for downloading and installing the JAR.Ambari Install or Upgrade of Falcon If you are using the Falcon service, the following steps must be performed on all Ambari server hosts on the cluster, prior to installing or upgrading to HDP 2.5.0

SYMPTOM: While integrating Storm 1.0.1 to Elastic Search 5.0.0, the following error is observed:Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/common/base/Preconditions at org.apache.storm.elasticsearch.common.EsConfig.<init>(EsConfig.java:62) at org.apache.storm.elasticsearch.common.EsConfig.<init>(EsConfig.java:49) at com.mz.pipeline.StreamToES5_1.main(StreamToES5_1.java:72) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.base.Preconditions Following were the maven dependencies:<dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-cor

SYMPTOM: For a Capacity Scheduler queue that specifies some groups in its acl_submit_applications property, a user who is not a member of any of those groups is still able to submit jobs to the queue. ROOT CAUSE: By default the root queue is allow-all, which results in all child queues defaulting to allow-all.The acl_submit_applications property is described as: The ACL which controls who can submit applications to the given queue. If the given user/group has necessary ACLs on the given queue or one of the parent queues in the hierarchy they can submit applications. ACLs for this property are inherited from the parent queue if not specified.SOLUTION: Set the root queue to deny-all, by entering a "space" for the value. Then set who to allow in the ACL for each child queue. For example:yarn.scheduler.capacity.root.acl_submit_applications= yarn.scheduler.capacity.root.default.acl_administer_jobs=appdevyarn.scheduler.capacity.root.default.acl_submit_applications=appdevyarn.scheduler.capac

PROBLEM DESCRIPTIONThe Oozie service check fails and the following error message is displayed in Ambari:stderr: /var/lib/ambari-agent/data/errors-12523.txtPython script has been killed due to timeout after waiting 300 secs There is no error in the stdout. The service check is terminated because the timeout (300 secs by default) is reached. CAUSEThis issue occurs when the time taken by Ambari to upload jar and workflow files to hdfs for Oozie service check takes longer than the timeout mentioned in the server settings.Example:Nodes in a cluster are configured on ipv4 proxy, which causes network slowness among nodes. Ambari uploads jar files and workflow files to the hdfs for Oozie service check, depending on network performance time required to upload these files exceeds 300 secs timeout.WORKAROUNDTo increase the timeout, find and update the timeout set in the metainfo.xml file located in Edit /var/lib/ambari-server/resources/common-services/OOZIE/your_version_number/metainfo.xml, RES

To configure Oozie workflow after enabling the ResourceManager HA, do the following: Find the YARN Resource Manager cluster id. To find the cluster id, Go to Ambari > YARN > configs Search for property yarn.resourcemanager.cluster-idAfter finding the cluster id, at the Job Property file, provide the YARN Resource Manager cluster id to the job _Tracker like the following:jobTracker= yarn-haSubmit the Oozie job using the modified job property file.

Does HBase replication guarantee consistency or snapshot-in-time?HBase replication works by propagating the WAL from the source cluster to the destination and is replayed asynchronously. Thus, it is eventually consistent.For snapshot functionality, consider taking a Snapshot on the source cluster and exporting that to the destination using the ExportSnapshot tool.Is there an option of enabling replication at the GLOBAL level rather than individually at the table level?HBase replication can be enabled at the table level or column family level onlyDoes enabling replication cause the table to be disabled?Check that the hbase.online.schema.update.enable in hbase-site.xmlis true. If so, then the table will not be disabled when turning on table replication.The default value is true so if the property is not set in hbase-site.xml, it is 'true'. Does HBase replication solve the problem of replicating a cluster which already has data to an empty cluster?Replication alone does not solve replica

Symptom:After adding additional configurations to Solr, the service fails with both 'no servers hosting shard' and 'OutOfMemoryError' errors. A restart will resolve the issue for some time and it will eventually hit the error again.Root Cause:In some versions, the default setting for max user processes for the infra-solr user is 1024, which may be set too low for certain use cases. Resolution:Increase the max user process user limit. This can be done by creating the file /etc/security/limits.d/infra-solr.conf on the solr node and setting the nproc property. For example if the new value chosen is 6000, the contents would be: infra-solr - nproc 6000

Symptom:The oldWALs directory stores WALs that are no longer needed for recovery. We have come across situations where the directory continually increases in size and the older WALs are not getting cleaned up.Root Cause:This can get filled up due to a bug where hbase.backup.enable is true by default in HDP-2.5.3Resolution:- Add the property to HBase Custom hbase-site:hbase.backup.enable=false- Restart HBase

ROOT CAUSE:kafka.metrics.reporters in Advanced Kafka-broker was pointing to Ganglia metrics reporter. kafka.metrics.reporters=kafka.ganglia.KafkaGangliaMetricsReporterRESOLUTION: Kafka.metrics.reporters is pointing to GangliaMetrics. It should be pointing to Ambari metrics like this: kafka.metrics.reporters=org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter You can modify the property via Ambari -> Kafka -> configs -> Advanced kafka-broker-> kafka.metrics.reporters and modify the property. Please save the changes and restart required services

SYMPTOM: Storm nimbus fails to come up after Ambari was upgraded to 2.4 version while on HDP 2.42016-11-22 13:29:43.066 [timer] b.s.d.nimbus [ERROR] Error when processing eventjava.lang.NullPointerException at clojure.lang.Numbers.ops(Numbers.java:961) ~[clojure-1.6.0.jar:?] at clojure.lang.Numbers.isZero(Numbers.java:90) ~[clojure-1.6.0.jar:?] at backtype.storm.util$partition_fixed.invoke(util.clj:892) ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?] at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?] at clojure.core$apply.invoke(core.clj:624) ~[clojure-1.6.0.jar:?] at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.6.0.jar:?] at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.6.0.jar:?] at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?] at clojure.core$partial$fn__4228.doInvoke(core.clj:2468) ~[clojure-1.6.0.jar:?] at clojure.lang.RestFn.invoke(RestFn.java:408)

We can configure multiple listeners by giving comma-separated list of URIs that Kafka will listen on. Please follow the steps below to implement this:1. Add the listeners as comma separated value in Ambari ->kafka->configs->listeners, for example:listeners=PLAINTEXT://myhost:6667, PLAINTEXTSASL://myhost:66682. Add ACL for 'Anonymous' user, because In PLAINTEXT connections user's identity is set to Anonymous. For example :$ bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=ambari-server:2181 --add --allow-principal User:Anonymous --producer --topic topic-oct3. Run the producer with security protocol set to PLAINTEXT to listen to PLAINTEXT and set it to PLAINTEXTSASL to listen to other listener, something like this:$ bin/kafka-console-producer.sh --broker-list ambari-server.support.com:6667 --topic topic-oct --security-protocol PLAINTEXT$ bin/kafka-console-producer.sh --broker-list ambari-server.support.com:6668 --t