Community Articles

If you would like NN1 to be active namenode, you can do it by stopping ZKFC of NN2 via ambari.After ~10 sec the failover should happend the the other NN become the active

Many customers are looking to use local indexes with HDP-2.6.1 to avoid some of the issues global secondary indexes like latency spikes in secondary index management codepath that have a way of cascading it to several region servers and grinding the entire cluster to a halt.Now HDP is using new local index implementation starting from 2.5. HDP 2.6.x has a majority of fixes for local indexes that exist in Apache Phoenix 4.8-4.9-4.10. In other words local indexes is ready and can be used in production.

This article tries to compare the data recovery period of accidentally deleted data in HDFS. We would compare two scenarios, 1. When trash is enabled. 2. When snapshot is enabled. Data Recovery from trash: When a data from HDFS is deleted, metadata in HDFS is updated to delete the file from the source folder. However, the blocks from the datanode is not immediately deleted. The trash folder in HDFS is updated with the file along with the directory from where it is deleted in the user's .trash folder. The deleted data could be recovered from the trash folder. Example:1. Existing data in HDFS.#hadoop fs -ls /tmp/test1.txt-rw-r--r-- 3 hdfs hdfs 4 2017-08-07 23:47 /tmp/test1.txt2. Deleted data in HDFS.#hadoop fs -rm /tmp/test1.txt17/08/07 23:52:13 INFO fs.TrashPolicyDefault: Moved: 'hdfs://vnn/tmp/test1.txt' to trash at: hdfs://vnn/user/hdfs/.Trash/Current/tmp/test1.txt3. Recovering a deleted data#hadoop fs -cp /user/hdfs/.Trash/Current/tmp/test1.txt /tmp/#hadoop fs -ls /

Please follow below steps to setup Oozie HA configuration with Kerberos environment.. Step 1: Configure mysql/oracle database for Oozie as HA configuration does not work with default embedded Derby Database..Please refer https://community.hortonworks.com/articles/183/moving-oozie-to-mysql-with-ambari.html for steps to migrate Oozie database..Step 2: Login to Ambari UI, goto hosts, select host on which you need to add additional Oozie server, Click on Add and select Oozie server..Please refer below screenshot, for e.g. I will add oozie server on kk3.hwxblr.com.Step 3: Setup Load balancerPlease refer this blogpost for setting up lightweight open source linux based load balancer..Step4: Configure Kerberos for your cluster if not already done.Please refer our blog for automated Kerberos configuration..Step 5: Login to Ambari UI and set below configuration parameters for Oozie service.oozie.zookeeper.connection.string=<zookeeper1>:2181,<zookeeper2>:2181,<zookeeper3>:2181o

Symtoms : NameNode HA states: active_namenodes =[], standby_namenodes =[], unknown_namenodes =[(u'nn1',Solution : Could be in order :1) Ambari is doing the timeout ( 5 sec is default ) and killing the process if the NN takes long to start you can change the value of the timeout in/var/lib/ambari-server/resources/common-services/HDFS/vXXXX/package/scripts/hdfs_namenode.pyFrom this:@retry(times=5, sleep_time=5, backoff_factor=2, err_class=Fail)To this:@retry(times=25, sleep_time=25, backoff_factor=2, err_class=Fail)if not enough to this:@retry(times=50, sleep_time=25, backoff_factor=2, err_class=Fail)2) Could be the Zookeeper not getting the status of the NNfor this you can try Restart zookeeper, if it's still not working , then try the followingCheck the content of the Znode ( hadoop-ha ), save the namespace of the NN and delete the content and restart the NN

Python Word Cloud Integrating existing Python libraries and scripts is very easy in Apache NiFi. I add the library for both version of Python I have on my system, while moving all new scripts to the 3.x branch.Install the library for both Python 2.7 and 3.5pip install wordcloud pip3 install wordcloud Example Usageecho "NiFi\nHadoop\nSpark\n" | wordcloud_cli.py --imagefile wordcloud.pngFor use in NiFi, I wrap my call with a shell script wc.sh echo $1 | tr " " "\n" | wordcloud_cli.py This will build a PNG for me that I can store in a file system or in HDFS, I updated the filename to add png at the end. This will take a parameter to a shell script (our Tweet) and convert it into words usable for a word cloud. You can use other sources or other methods of splitting words.I am pulling twitter messages, so I use ReplaceText to replace the flow file with ${msg} which is just the tweet.Then I execute the Python WordCloud CLI:ExampleReferences: https://amueller.github.io/word_cloud/auto_e

Technology: Python, TensorFlow, Apache Hive, MiniFi, NiFi, HDFS, WebHDFS, Zeppelin, SQL, Raspberry Pi, Pi Camera, S2S.Apache NiFi For Ingest of Images and TensorFlow Analysis from the Edge (Raspberry Pi 3)The Apache NiFi ingestion flow is straightforward. MiniFi sends us flow files over S2S from the RPI which consists of two types of messages. One is a JSON formatted file of metadata and TensorFlow analysis of an image. The second is the actual image captured. We route on the filename attribute to handle each file type appropriately. We send the image to HDFS for storage and retrieval via WebHDFS. The JSON I add a schema and a JSON content-type and split up the file. You see I have some non-JSON junk in there I want pulled out. Then I send my JSON record to QueryRecord to filter out any empty messages. This produces AVRO files from JSON which I convert to ORC and store in HDFS. From there it's easy to query my new deep learning produced multimedia data via standard

Goal: Demonstrate how to change the database location in HDFS and MetastoreThere are circumstances wherein we can consider moving the database location. By default, the location for default and custom databases is defined within the value of hive.metastore.warehouse.dir, which is /apps/hive/warehouse.Here are the illustrated steps to change a custom database location, for instance "dummy.db", along with the contents of the database.Verify the details of the database we would like to move to a new location[hive@xlautomation-2 ~]$ beeline -u "jdbc:hive2://xlautomation-2.h.c:10000/default;principal=hive/xlautomation-2.h.c@H.C"0: jdbc:hive2://xlautomation-2.h.c:10000/defa> create database dummy;No rows affected (0.394 seconds)0: jdbc:hive2://xlautomation-2.h.c:10000/defa> describe database dummy;+----------+----------+--------------------------------------------------------------+-------------+-------------+-------------+--+| db_name | comment | location

Short DescriptionHow to setup popular SQL development tools, like DbVisualizer, to access HDP Hive in secured network zone via Knox.ArticleMost customers setup Hadoop/Hive cluster in secured network Zone. There's no direct network connection between office and Hadoop network. So Knox is always setup to proxy Hive connections from office network. This article will setup DbVisualizer to connect to Knox Hive URL step by step.Download Hive Driver for HDP2.6.1Download hive-jdbc.jar https://github.com/timveil/hive-jdbc-uber-jar/releasesThe latest for HDP2.6.1 is https://github.com/timveil/hive-jdbc-uber-jar/releases/download/v1.6-2.6.1/hive-jdbc-uber-2.6.1.0-129.jarCreate a new Database Driver use the hive-jdbc.jarURL Format: jdbc:hive2://<server>:<port10000>/<database>The URL Format is for standard hive2 jdbc, will change later in actual connection for Knox.Create a new Connection with this new DriverUse Database URL, and edit the Database URL asjdbc:hive2://<KNOX_NODE_

ENVIRONMENT: HDP 2.5 with RM in HA.SYMPTOMS: gaurav@g1:~> hadoop jar /usr/hdp/2.5.0.0-1245/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 5 8 Number of Maps = 5amples per Map = 8 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Starting Job 11/02/16 12:20:01 INFO impl.TimelineClientImpl: Timeline service address: http://g1.openstacklocal:8188/ws/v1/timeline/11/02/16 12:20:02 INFO client.AHSProxy: Connecting to Application History server at /0.0.0.0:10200 11/02/16 12:20:05 WARN ipc.Client: Failed to connect to server: g2.openstacklocal:8032: retries get failed due to exceeded maximum allowed retries number: 0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)ROOT CAUSE: This a known behavior reported in internal jira BUG-65968.This is not a problem but an expected behavior.No matter which RM is currently active, the application would always try to connect to the rm1, then rm2. Please

Creating and running Temporary functions are discouraged while running a query on LLAP because of security reason since many users are sharing same instances of LLAP, it can create a conflict but still, you can create temp functions using add jar and hive.llap.execution.mode=auto. with exclusive llap execution mode(hive.llap.execution.mode=only) you will run into the ClassNotFoundException, hive.llap.execution.mode=auto will allow some part of query(map tasks) to run in the tez container. Here are steps to create a custom permanent function in LLAP(steps are tested on HDP-260) 1. create a jar for UDF function (in this case I am using simple udf):git clone https://github.com/rajkrrsingh/SampleCode mvn clean package2. upload the target/SampleCode.jar to the node where HSI is running(in my case I have copied it to /tmp directory) 3. add jar to hive_aux_jars (goto Ambari--> hive --> config --> hive-interactive-env template)export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:/tmp/SampleC

Caution: Running bad queries against the AMS hbase tables can crash the AMS collector PID due to load. Use it for debugging purpose.To connect to AMS Hbase instance which is running in distributed mode.cd /usr/lib/ambari-metrics-collector/bin./sqlline.py localhost:2181:/ams-hbase-secureTo get the correct znode, get the value of "zookeeper.znode.parent" from AMS collector configs.

Log4j settingsAmbari uses sets of properties called Log4j properties to control logging activities for each service running in your Hadoop cluster. Initial, default values for each property reside in a <service_name>-log4j template file.To access the default Log4j settings for a service; in Ambari Web, you can browse to <Service_name> > Configs > Advanced <service_name>-log4j.Often, user might want to configure the number of backup log files or limit the maximum size of a log file. Prior to Ambari-2.5, this had to be done by manually searching for the property in the log4j template Text Area and modifying it. This is quite inconvenient.Ambari-2.4 and beforeAmbari-2.5With Ambari-2.5, a simplified log rotation configuration is provided i.e Log4j properties and values that limit the size and number of backup log files for each service now appear above the log4j template file as separate fields.Steps1.Edit the values for the <service_name> backup file size and

Problem:I have been trying to configure Zeppelin's JDBC interpreter to work with our Phoenix servers, but am getting an error when running queries. The JDBC interpreter works fine for Hive and MySQL. Running this:%jdbc(phoenix)select * from <table>I am gettingorg.apache.zeppelin.interpreter.InterpreterException: nullorg.apache.phoenix.exception.PhoenixIOException: Failed after attempts=1, exceptions:Thu Jul 20 08:27:49 BST 2017, RpcRetryingCaller{globalStartTime=1500535669736, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipeat org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:416)at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:564)at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:692)at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)at org.apache.zeppelin.interprete

Hbase indexing to Solr with HDP Search in HDP 2.3 Background:The HBase Indexer provides the ability to stream events from HBase to Solr for near real time searching. The HBase indexer is included with HDPSearch as an additional service. The indexer works by acting as an HBase replication sink. As updates are written to HBase, the events are asynchronously replicated to the HBase Indexer processes, which in turn creates Solr documents and pushes them to Solr. References:https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Jobs.html#_hbase-indexerhttps://github.com/NGDATA/hbase-indexer/wiki/TutorialSteps Download and start HDP 2.3 sandbox VM which comes with LW HDP search installed (under /opt/lucidworks-hdpsearch) and run below to ensure no log files owned by root remainchown -R solr:solr /opt/lucidworks-hdpsearch/solrIf running on an Ambari installed HDP 2.3 cluster (instead of sandbox), run the below to install HDPsearch and setup the user dir in HDFS:yum install -y lucidworks-hd

A series of examples and flow files:https://github.com/xmlking/nifi-examplesNiFi ExamplesApache NiFi example flows.collect-stream-logsThis flow shows workflow for log collection, aggregation, store and display.Ingest logs from folders.Listen for syslogs on UDP port.Merge syslogs and drop-in logs and persist merged logs to Solr for historical search.Dashboard: stream real-time log events to dashboard and enable cross-filter search on historical logs data.csv-to-jsonThis flow shows how to convert a CSV entry to a JSON document using ExtractText and ReplaceText.decompressionThis flow demonstrates taking an archive that is created with several levels of compression and then continuously decompressing it using a loop until the archived file is extracted out.http-get-routehis flow pulls from a web service (example is nifi itself), extracts text from a specific section, makes a routing decision on that extracted value, prepares to write to disk using PutFile.invoke-http-routeThis flow demonst

Affected versions: 1.3.xSymptoms: Bundles are completely empty even if the status is completed or ready.Reason:This issue will typically happen in case of some unexpected scenarios during installation or upgrade. Some of the stale configurations which were not removed completely, conflict with the new configurations.Solution: Verify whether /etc/hst/conf/capture_levels.json and /etc/hst/conf/anonymization_rules.json are latest as per installed version.If the above config files are as expected, then verify whether the following config is correct in Ambari using configs.sh:/var/lib/ambari-server/resources/scripts/configs.sh –u admin –p admin –port 8080 get ambarihost clustername capture-levels capture-levels-contentIf there are multiple entries, then delete the old one using command:/var/lib/ambari-server/resources/scripts/configs.sh –u admin –p admin –port 8080 get ambarihost clustername capture-levels capture-levels-content

In large clusters , sometimes restarting Namenode or a secondary namenode will fail and Ambari will keep trying multiple times then fails.One thing can be done quickly is to increase the timeouts of Ambari from 5s to 25s ( or up to 50s )In/var/lib/ambari-server/resources/common-services/HDFS/XXX-VERSION-XXX/package/scripts/hdfs_namenode.pyFrom this: @retry(times=5, sleep_time=5, backoff_factor=2, err_class=Fail)To this: @retry(times=25, sleep_time=25, backoff_factor=2, err_class=Fail)If it still fail, you can try @retry(times=50, sleep_time=50, backoff_factor=2, err_class=Fail)One of the root causes of this maybe SOLR audit logs ( from Ambari Infra ) when creating huge logs that needs to be written to hdfs.Restart Ambari serverYou can clear the logs of NN and SNN here : /var/log/hadoop/hdfs/audit/solr/spoolBecareful on deleting only on Standby NN - then do a failover to delete from the other server. do not delete logs while the namenode is active.

In large clusters , sometimes restarting Namenode or a secondary namenode will fail and Ambari will keep trying mltiple times then fail.One thing can be done quickly is to increase the timeouts of Ambari from 5s to 25s.In/var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.pyFrom this: @retry(times=5, sleep_time=5, backoff_factor=2, err_class=Fail)To this: @retry(times=25, sleep_time=25, backoff_factor=2, err_class=Fail)If it still fail, you can try @retry(times=50, sleep_time=50, backoff_factor=2, err_class=Fail)One of the root causes of this maybe SOLR audit logs ( from Ambari Infra ) when creating huge logs that needs to be written to hdfs.You can clear the logs of NN and SNN here : /var/log/hadoop/hdfs/audit/solr/spoolBecareful on deleting only on Standby NN - then do a failover to delete from the other server. do not delete logs while the namenode is active.

ISSUE: Spark Job fails with "java.lang.LinkageError: ClassCastException: attempting to castjar:file" because of a conflict between RuntimeDelegate from Jersey in yarn client libs and the copy in spark's assembly jar.ERROR:17/05/02 17:44:25 ERROR ApplicationMaster: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classjava.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.

Consolidating below some of the errors thrown by Spark Thrift Server during SQL execution, that could be worked around by configuring certain parameters of spark-thrift-sparkconf.conf and hive-site.xml Error 1: Join condition is missing or trivial.Use the CROSS JOIN syntax to allow cartesian products between these relations.; Resolution: spark.sql.crossjoin.enabled: true Error 2: Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "eval(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" grows beyond 64 KB Resolution: spark.sql.codegen.wholeStage : false Error 3: java.lang.OutOfMemoryError: Java heap space Resolution: spark.driver.memory : 10g <to a higher-value> spark.sql.ui.retainedExecutions: 5 <to some lower-value> Error 4:org.apache.spark.SparkException: Exception thrown in awaitResult: (state=,code=0)Resolution: hive.metastore.try.direct.sql:

Issue:Running the below sqoop on a client machine might lead to the reported errorsqoop export -Dmapreduce.map.log.level=DEBUG -Dteradata.db.output.fastload.socket.port=8678 \-Dteradata.db.output.method=internal.fastload \--connect jdbc:teradata://mrt1.openstacklocal/Database=testdb \--connection-manager org.apache.sqoop.teradata.TeradataConnManager --username COOXP -P \--export-dir /user/karthick/fastload_test \--table test_table --input-fields-terminated-by ',' \--null-string '\N' --null-non-string '\N' --num-mappers 5 INFO mapreduce.Job: Task Id : attempt_1497611885470_0832_m_000000_0, Status : FAILEDError: com.teradata.connector.common.exception.ConnectorException: java.net.SocketException: Socket is not connected at java.net.Socket.getInputStream(Socket.java:905) at com.teradata.connector.teradata.TeradataInternalFastloadOutputFormat.getRecordWriter(TeradataInternalFastloadOutputFormat.java:356) at com.teradata.connector.common.ConnectorOutputFormat$ConnectorF

OverviewThe following versions of Apache Kafka have been incorporated in HDP 2.2.0 to 2.6.1: 0.8.1, 0.8.2, 0.9.0, 0.10.0, 0.10.1. Apache Kafka is now at 0.11.Hortonworks is working to make Kafka easier for enterprises to use. New focus areas include creation of a Kafka Admin Panel to create/delete topics and manage user permissions, easier and safer distribution of security tokens and support for multiple ways of publishing/consuming data via a Kafka REST server/API. Here are a few areas of strong contribution:Operations:Rack awareness for Increased resilience and availability such that replicas are isolated so they are guaranteed to span multiple racks or availability zones.Automated replica leader election for automated, even distribution of leaders in a cluster capability by detecting uneven distribution with some brokers serving more data compared to others and makes adjustments.Message Timestamps so every message in Kafka now has a timestamp field that indicates the time at which

With HDF 2.x, Ambari can be used to deploy a NiFi cluster. Lets say you deployed a 2 node cluster and want to go back at a later time and add an additional NiFi node to the cluster. While the process is very straight forward when your NiFi cluster has been setup non-secure (http), the same is not true if your existing NiFi cluster has been secured (https).Below you will see an existing 2 node secured NiFi cluster that was installed via Ambari:STEP 1: Add new host through Ambari. You can skip this step if the host you want to install the additional NiFi node on is already managed by your Ambari.STEP 2: Under "Hosts" in Ambari click on the host form the list where you want to install the new NiFi node.The NiFi component will be in a "stopped" state after it is installed on this new host. *** DO NOT START NIFI YET ON NEW HOST OR IT WILL FAIL TO JOIN CLUSTER. ***STEP 3: (This step only applies if NiFi's file based authorizer is being used) Before starting this new node we need to clea

If cluster size is large and it is more than 1 year then Ambari gets slows down bit - so it is recommended to purge historical operational data.Note: You would loose upgrade history as well - Ambari does not use the upgrade history in any way. Note: (Updated in August 2017) :From Ambari 2.5.x there is a product utility to do the clean up - "db-cleanup"Here is list of queries to delete more than 1 month operational data. please do this with caution.Take a database backup before attempting to perform this.CREATE TEMPORARY TABLE IF NOT EXISTS tmp_request_id AS SELECT MAX(request_id) AS request_id FROM request WHERE create_time <= (SELECT (UNIX_TIMESTAMP(NOW()) - 2678400) * 1000 as epoch_1_month_ago_times_1000);CREATE TEMPORARY TABLE IF NOT EXISTS tmp_task_id AS SELECT MAX(task_id) AS task_id FROM host_role_command WHERE request_id <= (SELECT request_id FROM tmp_request_id);CREATE TEMPORARY TABLE IF NOT EXISTS tmp_upgrade_ids AS SELECT upgrade_id FROM upgrade WHERE request_id <