Resources

Many organizations still ask the question, “Can I run BI (Business Intelligence) workloads on Hadoop?” These workloads range from short, low-latency ad-hoc queries to canned or operational reporting. The primary concerns center around user experience. Will a query take too long to return an answer? How quickly can I change my mind with a report and drill down other dimensional attributes? For almost 20 years vendors have engineered highly customized solutions to solve these problems. Many times these solutions require fine-tuned appliances that tightly integrate hardware and software in order to squeeze out every last drop of performance. The challenges with these solutions are mainly around cost and maintenance. These solutions become cost-prohibitive at scale and require large teams to manage and operate. The ideal solution is one that affordably scales but retains the same performance advantages as your appliance. Your analysts should not see the difference between the costly a

Very well know way to handle JSON is to use JSON SerDe which originated from hcatalog. There is another interesting rather unconventional method to handle JSON data in HIVE. json_tuple and LATERAL VIEW. Table here only has one column that loads JSON data as a single string. json_tuple() is a User defined Table Function ( UDTF ) introduced in Hive 0.7. It takes a set of names (keys) and a JSON string, and returns a tuple of values using one function.In the words of ASF "A lateral view first applies the UDTF to each row of base table and then joins resulting output rows to the input rows to form a virtual table having the supplied table alias".Consider below is sample data line:{"user":{"location":"","id":1514008171,"name":"Auzzie Jet","screenname":"metalheadgrunge","geoenabled":false},"tweetmessage":"Anthrax - Black - Sunglasses hell yah\n http://t.co/qCNjba57Dm","createddate":"2013-06-20T12:08:44","geolocation":null}Table creation:CREATE EXTERNAL table tweets{ tweet STRING }LOAD dat

In the following article, Ambari Views Server is the Standalone Ambari Server used for hosting Views and Ambari Server is the Operational Ambari Server which manages a Hadoop ClusterBefore Ambari 2.4, when Ambari Views Servers are setup, the only way to configure views was to use ‘Custom Configuration’. In this method details had to be manually entered for several fields. For example, to configure Files View, all the entries as described in Ambari Views Documentaion for 2.2 had to be made.This is made easier in Ambari 2.4 with ‘Remote Cluster Configuration’ . The remote cluster configuration is used to set up an Ambari Views Server to host views for a remote Ambari server, regardless of the cluster configuration.Once a remote cluster configuration is set, all it takes to create a new view is to select the 'Remote Cluster' option in 'Cluster configuration' and select a cluster name in create view wizard. How to define a new remote cluster configuration?Note: If remote Ambari server is e

Previous article demonstrated how to use nifi rest api to create a template. The objective for this article is to demonstrate how to use NiFi rest api to change a work flow on a secured cluster real time without service disruption or downtime.Use CaseA process group named CICD is a active data flow (data is flowing through it) and a new processor will be added at the end of a flow without application disruption or downtime. Comments in the picture above.StepsCreate Access TokenCreate Client IDFetch Process Group attributes (process group which the changes will be applied to)Create LogAttribute ProcessorFetch the UUID of the UpdateAttribute processor Why? Data will flow from this existing processor into newly added LogAttribute Processor Link LogAttribute & UpdateAttribute via success relationshipUpdate LogAttribute to terminate on success relationshipSteps 1 & 2 & 3These steps are described here so no reason to copy and paste.Step 4 - Create LogAttribute Processorpayload

If you would like NN1 to be active namenode, you can do it by stopping ZKFC of NN2 via ambari.After ~10 sec the failover should happend the the other NN become the active

Many customers are looking to use local indexes with HDP-2.6.1 to avoid some of the issues global secondary indexes like latency spikes in secondary index management codepath that have a way of cascading it to several region servers and grinding the entire cluster to a halt.Now HDP is using new local index implementation starting from 2.5. HDP 2.6.x has a majority of fixes for local indexes that exist in Apache Phoenix 4.8-4.9-4.10. In other words local indexes is ready and can be used in production.

This article tries to compare the data recovery period of accidentally deleted data in HDFS. We would compare two scenarios, 1. When trash is enabled. 2. When snapshot is enabled. Data Recovery from trash: When a data from HDFS is deleted, metadata in HDFS is updated to delete the file from the source folder. However, the blocks from the datanode is not immediately deleted. The trash folder in HDFS is updated with the file along with the directory from where it is deleted in the user's .trash folder. The deleted data could be recovered from the trash folder. Example:1. Existing data in HDFS.#hadoop fs -ls /tmp/test1.txt-rw-r--r-- 3 hdfs hdfs 4 2017-08-07 23:47 /tmp/test1.txt2. Deleted data in HDFS.#hadoop fs -rm /tmp/test1.txt17/08/07 23:52:13 INFO fs.TrashPolicyDefault: Moved: 'hdfs://vnn/tmp/test1.txt' to trash at: hdfs://vnn/user/hdfs/.Trash/Current/tmp/test1.txt3. Recovering a deleted data#hadoop fs -cp /user/hdfs/.Trash/Current/tmp/test1.txt /tmp/#hadoop fs -ls /

Please follow below steps to setup Oozie HA configuration with Kerberos environment.. Step 1: Configure mysql/oracle database for Oozie as HA configuration does not work with default embedded Derby Database..Please refer https://community.hortonworks.com/articles/183/moving-oozie-to-mysql-with-ambari.html for steps to migrate Oozie database..Step 2: Login to Ambari UI, goto hosts, select host on which you need to add additional Oozie server, Click on Add and select Oozie server..Please refer below screenshot, for e.g. I will add oozie server on kk3.hwxblr.com.Step 3: Setup Load balancerPlease refer this blogpost for setting up lightweight open source linux based load balancer..Step4: Configure Kerberos for your cluster if not already done.Please refer our blog for automated Kerberos configuration..Step 5: Login to Ambari UI and set below configuration parameters for Oozie service.oozie.zookeeper.connection.string=<zookeeper1>:2181,<zookeeper2>:2181,<zookeeper3>:2181o

Symtoms : NameNode HA states: active_namenodes =[], standby_namenodes =[], unknown_namenodes =[(u'nn1',Solution : Could be in order :1) Ambari is doing the timeout ( 5 sec is default ) and killing the process if the NN takes long to start you can change the value of the timeout in/var/lib/ambari-server/resources/common-services/HDFS/vXXXX/package/scripts/hdfs_namenode.pyFrom this:@retry(times=5, sleep_time=5, backoff_factor=2, err_class=Fail)To this:@retry(times=25, sleep_time=25, backoff_factor=2, err_class=Fail)if not enough to this:@retry(times=50, sleep_time=25, backoff_factor=2, err_class=Fail)2) Could be the Zookeeper not getting the status of the NNfor this you can try Restart zookeeper, if it's still not working , then try the followingCheck the content of the Znode ( hadoop-ha ), save the namespace of the NN and delete the content and restart the NN

Python Word Cloud Integrating existing Python libraries and scripts is very easy in Apache NiFi. I add the library for both version of Python I have on my system, while moving all new scripts to the 3.x branch.Install the library for both Python 2.7 and 3.5pip install wordcloud pip3 install wordcloud Example Usageecho "NiFi\nHadoop\nSpark\n" | wordcloud_cli.py --imagefile wordcloud.pngFor use in NiFi, I wrap my call with a shell script wc.sh echo $1 | tr " " "\n" | wordcloud_cli.py This will build a PNG for me that I can store in a file system or in HDFS, I updated the filename to add png at the end. This will take a parameter to a shell script (our Tweet) and convert it into words usable for a word cloud. You can use other sources or other methods of splitting words.I am pulling twitter messages, so I use ReplaceText to replace the flow file with ${msg} which is just the tweet.Then I execute the Python WordCloud CLI:ExampleReferences: https://amueller.github.io/word_cloud/auto_e

Technology: Python, TensorFlow, Apache Hive, MiniFi, NiFi, HDFS, WebHDFS, Zeppelin, SQL, Raspberry Pi, Pi Camera, S2S.Apache NiFi For Ingest of Images and TensorFlow Analysis from the Edge (Raspberry Pi 3)The Apache NiFi ingestion flow is straightforward. MiniFi sends us flow files over S2S from the RPI which consists of two types of messages. One is a JSON formatted file of metadata and TensorFlow analysis of an image. The second is the actual image captured. We route on the filename attribute to handle each file type appropriately. We send the image to HDFS for storage and retrieval via WebHDFS. The JSON I add a schema and a JSON content-type and split up the file. You see I have some non-JSON junk in there I want pulled out. Then I send my JSON record to QueryRecord to filter out any empty messages. This produces AVRO files from JSON which I convert to ORC and store in HDFS. From there it's easy to query my new deep learning produced multimedia data via standard

Goal: Demonstrate how to change the database location in HDFS and MetastoreThere are circumstances wherein we can consider moving the database location. By default, the location for default and custom databases is defined within the value of hive.metastore.warehouse.dir, which is /apps/hive/warehouse.Here are the illustrated steps to change a custom database location, for instance "dummy.db", along with the contents of the database.Verify the details of the database we would like to move to a new location[hive@xlautomation-2 ~]$ beeline -u "jdbc:hive2://xlautomation-2.h.c:10000/default;principal=hive/xlautomation-2.h.c@H.C"0: jdbc:hive2://xlautomation-2.h.c:10000/defa> create database dummy;No rows affected (0.394 seconds)0: jdbc:hive2://xlautomation-2.h.c:10000/defa> describe database dummy;+----------+----------+--------------------------------------------------------------+-------------+-------------+-------------+--+| db_name | comment | location

Short DescriptionHow to setup popular SQL development tools, like DbVisualizer, to access HDP Hive in secured network zone via Knox.ArticleMost customers setup Hadoop/Hive cluster in secured network Zone. There's no direct network connection between office and Hadoop network. So Knox is always setup to proxy Hive connections from office network. This article will setup DbVisualizer to connect to Knox Hive URL step by step.Download Hive Driver for HDP2.6.1Download hive-jdbc.jar https://github.com/timveil/hive-jdbc-uber-jar/releasesThe latest for HDP2.6.1 is https://github.com/timveil/hive-jdbc-uber-jar/releases/download/v1.6-2.6.1/hive-jdbc-uber-2.6.1.0-129.jarCreate a new Database Driver use the hive-jdbc.jarURL Format: jdbc:hive2://<server>:<port10000>/<database>The URL Format is for standard hive2 jdbc, will change later in actual connection for Knox.Create a new Connection with this new DriverUse Database URL, and edit the Database URL asjdbc:hive2://<KNOX_NODE_

ENVIRONMENT: HDP 2.5 with RM in HA.SYMPTOMS: gaurav@g1:~> hadoop jar /usr/hdp/2.5.0.0-1245/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 5 8 Number of Maps = 5amples per Map = 8 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Starting Job 11/02/16 12:20:01 INFO impl.TimelineClientImpl: Timeline service address: http://g1.openstacklocal:8188/ws/v1/timeline/11/02/16 12:20:02 INFO client.AHSProxy: Connecting to Application History server at /0.0.0.0:10200 11/02/16 12:20:05 WARN ipc.Client: Failed to connect to server: g2.openstacklocal:8032: retries get failed due to exceeded maximum allowed retries number: 0 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)ROOT CAUSE: This a known behavior reported in internal jira BUG-65968.This is not a problem but an expected behavior.No matter which RM is currently active, the application would always try to connect to the rm1, then rm2. Please

Creating and running Temporary functions are discouraged while running a query on LLAP because of security reason since many users are sharing same instances of LLAP, it can create a conflict but still, you can create temp functions using add jar and hive.llap.execution.mode=auto. with exclusive llap execution mode(hive.llap.execution.mode=only) you will run into the ClassNotFoundException, hive.llap.execution.mode=auto will allow some part of query(map tasks) to run in the tez container. Here are steps to create a custom permanent function in LLAP(steps are tested on HDP-260) 1. create a jar for UDF function (in this case I am using simple udf):git clone https://github.com/rajkrrsingh/SampleCode mvn clean package2. upload the target/SampleCode.jar to the node where HSI is running(in my case I have copied it to /tmp directory) 3. add jar to hive_aux_jars (goto Ambari--> hive --> config --> hive-interactive-env template)export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:/tmp/SampleC

Caution: Running bad queries against the AMS hbase tables can crash the AMS collector PID due to load. Use it for debugging purpose.To connect to AMS Hbase instance which is running in distributed mode.cd /usr/lib/ambari-metrics-collector/bin./sqlline.py localhost:2181:/ams-hbase-secureTo get the correct znode, get the value of "zookeeper.znode.parent" from AMS collector configs.

Log4j settingsAmbari uses sets of properties called Log4j properties to control logging activities for each service running in your Hadoop cluster. Initial, default values for each property reside in a <service_name>-log4j template file.To access the default Log4j settings for a service; in Ambari Web, you can browse to <Service_name> > Configs > Advanced <service_name>-log4j.Often, user might want to configure the number of backup log files or limit the maximum size of a log file. Prior to Ambari-2.5, this had to be done by manually searching for the property in the log4j template Text Area and modifying it. This is quite inconvenient.Ambari-2.4 and beforeAmbari-2.5With Ambari-2.5, a simplified log rotation configuration is provided i.e Log4j properties and values that limit the size and number of backup log files for each service now appear above the log4j template file as separate fields.Steps1.Edit the values for the <service_name> backup file size and

Problem:I have been trying to configure Zeppelin's JDBC interpreter to work with our Phoenix servers, but am getting an error when running queries. The JDBC interpreter works fine for Hive and MySQL. Running this:%jdbc(phoenix)select * from <table>I am gettingorg.apache.zeppelin.interpreter.InterpreterException: nullorg.apache.phoenix.exception.PhoenixIOException: Failed after attempts=1, exceptions:Thu Jul 20 08:27:49 BST 2017, RpcRetryingCaller{globalStartTime=1500535669736, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipeat org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:416)at org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:564)at org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:692)at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)at org.apache.zeppelin.interprete

Hbase indexing to Solr with HDP Search in HDP 2.3 Background:The HBase Indexer provides the ability to stream events from HBase to Solr for near real time searching. The HBase indexer is included with HDPSearch as an additional service. The indexer works by acting as an HBase replication sink. As updates are written to HBase, the events are asynchronously replicated to the HBase Indexer processes, which in turn creates Solr documents and pushes them to Solr. References:https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Jobs.html#_hbase-indexerhttps://github.com/NGDATA/hbase-indexer/wiki/TutorialSteps Download and start HDP 2.3 sandbox VM which comes with LW HDP search installed (under /opt/lucidworks-hdpsearch) and run below to ensure no log files owned by root remainchown -R solr:solr /opt/lucidworks-hdpsearch/solrIf running on an Ambari installed HDP 2.3 cluster (instead of sandbox), run the below to install HDPsearch and setup the user dir in HDFS:yum install -y lucidworks-hd

A series of examples and flow files:https://github.com/xmlking/nifi-examplesNiFi ExamplesApache NiFi example flows.collect-stream-logsThis flow shows workflow for log collection, aggregation, store and display.Ingest logs from folders.Listen for syslogs on UDP port.Merge syslogs and drop-in logs and persist merged logs to Solr for historical search.Dashboard: stream real-time log events to dashboard and enable cross-filter search on historical logs data.csv-to-jsonThis flow shows how to convert a CSV entry to a JSON document using ExtractText and ReplaceText.decompressionThis flow demonstrates taking an archive that is created with several levels of compression and then continuously decompressing it using a loop until the archived file is extracted out.http-get-routehis flow pulls from a web service (example is nifi itself), extracts text from a specific section, makes a routing decision on that extracted value, prepares to write to disk using PutFile.invoke-http-routeThis flow demonst

Affected versions: 1.3.xSymptoms: Bundles are completely empty even if the status is completed or ready.Reason:This issue will typically happen in case of some unexpected scenarios during installation or upgrade. Some of the stale configurations which were not removed completely, conflict with the new configurations.Solution: Verify whether /etc/hst/conf/capture_levels.json and /etc/hst/conf/anonymization_rules.json are latest as per installed version.If the above config files are as expected, then verify whether the following config is correct in Ambari using configs.sh:/var/lib/ambari-server/resources/scripts/configs.sh –u admin –p admin –port 8080 get ambarihost clustername capture-levels capture-levels-contentIf there are multiple entries, then delete the old one using command:/var/lib/ambari-server/resources/scripts/configs.sh –u admin –p admin –port 8080 get ambarihost clustername capture-levels capture-levels-content

In large clusters , sometimes restarting Namenode or a secondary namenode will fail and Ambari will keep trying multiple times then fails.One thing can be done quickly is to increase the timeouts of Ambari from 5s to 25s ( or up to 50s )In/var/lib/ambari-server/resources/common-services/HDFS/XXX-VERSION-XXX/package/scripts/hdfs_namenode.pyFrom this: @retry(times=5, sleep_time=5, backoff_factor=2, err_class=Fail)To this: @retry(times=25, sleep_time=25, backoff_factor=2, err_class=Fail)If it still fail, you can try @retry(times=50, sleep_time=50, backoff_factor=2, err_class=Fail)One of the root causes of this maybe SOLR audit logs ( from Ambari Infra ) when creating huge logs that needs to be written to hdfs.Restart Ambari serverYou can clear the logs of NN and SNN here : /var/log/hadoop/hdfs/audit/solr/spoolBecareful on deleting only on Standby NN - then do a failover to delete from the other server. do not delete logs while the namenode is active.

In large clusters , sometimes restarting Namenode or a secondary namenode will fail and Ambari will keep trying mltiple times then fail.One thing can be done quickly is to increase the timeouts of Ambari from 5s to 25s.In/var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.pyFrom this: @retry(times=5, sleep_time=5, backoff_factor=2, err_class=Fail)To this: @retry(times=25, sleep_time=25, backoff_factor=2, err_class=Fail)If it still fail, you can try @retry(times=50, sleep_time=50, backoff_factor=2, err_class=Fail)One of the root causes of this maybe SOLR audit logs ( from Ambari Infra ) when creating huge logs that needs to be written to hdfs.You can clear the logs of NN and SNN here : /var/log/hadoop/hdfs/audit/solr/spoolBecareful on deleting only on Standby NN - then do a failover to delete from the other server. do not delete logs while the namenode is active.

ISSUE: Spark Job fails with "java.lang.LinkageError: ClassCastException: attempting to castjar:file" because of a conflict between RuntimeDelegate from Jersey in yarn client libs and the copy in spark's assembly jar.ERROR:17/05/02 17:44:25 ERROR ApplicationMaster: User class threw exception: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classjava.lang.LinkageError: ClassCastException: attempting to castjar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar!/javax/ws/rs/ext/RuntimeDelegate.classtojar:file:/u/applic/data/hdfs7/hadoop/yarn/local/filecache/469/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.

Consolidating below some of the errors thrown by Spark Thrift Server during SQL execution, that could be worked around by configuring certain parameters of spark-thrift-sparkconf.conf and hive-site.xml Error 1: Join condition is missing or trivial.Use the CROSS JOIN syntax to allow cartesian products between these relations.; Resolution: spark.sql.crossjoin.enabled: true Error 2: Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method "eval(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" grows beyond 64 KB Resolution: spark.sql.codegen.wholeStage : false Error 3: java.lang.OutOfMemoryError: Java heap space Resolution: spark.driver.memory : 10g <to a higher-value> spark.sql.ui.retainedExecutions: 5 <to some lower-value> Error 4:org.apache.spark.SparkException: Exception thrown in awaitResult: (state=,code=0)Resolution: hive.metastore.try.direct.sql: