Community Articles

Objective This tutorial demonstrates how to use the PutElasticsearchHttpRecord processor to easily put data from a CSV file into Elasticsearch. Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0 Elasticsearch 2.3.3PublishElasticsearchHttpRecord (CSVReader)Demo ConfigurationElasticsearch For my environment, I had Elasticsearch 2.3.3 installed. Start Elasticsearch and assign cluster and node names: ./elasticsearch --cluster.name elasticsearch --node.name hcc I like to use Elastic HQ to manage/monitor my cluster: Initial Flow One of the great things about the record-oriented functionality in NiFi is the ability to re-use Record Readers and Writers. In conjunction with the Record processors, it is quick and easy to change data formats and data de

The parent article discusses how to access the ams-hbase instance using Phoenix client when it is a standalone environment.This article is an extension to that as I list the steps to access ams-hbase instance using Phoenix client when Zookeeper is installed on the cluster.As described in the parent article, login to AMS Collector host machine:1. Check the /etc/ams-hbase/conf/hbase-site.xml2. From the above file we pick hbase.zookeeper.quorum , hbase.zookeeper.property.clientPort and zookeeper.znode.parent values3. Goto cd /usr/lib/ambari-metrics-collector/bin4. Invoke the client with the connection url as shown below by substituting the values: ./sqlline.py hbase.zookeeper.quorum : hbase.zookeeper.property.clientPort 😕 zookeeper.znode.parentExample ./sqlline.py myzkqrm.com:61181:/ams-hbase-secure

It is expected in large clusters where node count ranges to few hundreds, the master services tend to be busy. One such master service is Namenode. Some of the critical activities that NN does includes, 1. Addressing client requests which includes verifying proper permissions, auth checks for HDFS resources.2. Constant block report monitoring from all the Datanodes.3. Updating the service and audit logs.are to name a few. In certain situations when there are rogue applications which tries to access multiple resources in HDFS or a data ingestion that is trying to load high data volumes, NN tends to be very busy. In such situations and cluster like these NN FSImage tends to be in $$GB. Hence, operations such as checkpointing would consume considerable bandwidth across the two Namenodes. Hence, high volume of edits sync along with loggings would cause high disk utilization which can lead to NameNode instability. Hence, it is recommended to have dedicated disks for service logs and edit lo

Introduction & Motivation Apache Knox acts as a proxy between Hadoop services and their consumers. Knox topology files define the mapping between requested services and their respective endpoints. To make Knox aware of new mappings, a Knox administrator has to determine the endpoints for every service to be proxied, and add them to a topology file (XML). When working with an Ambari-managed cluster, the Knox administrator has the advantage of Ambari's knowledge of the cluster details. However, navigating the Ambari UI to locate the correct pieces of information to assemble these service endpoint URLs can be challenging, and having a human type them into an XML file introduces potential for errors. Fortunately, Ambari's API can be leveraged to automate the determination of these service endpoint URLs and, ultimately, the generation of Knox topology files.Knox Topology Files A topology file has to include a service entry for every Hadoop service to be proxied. Listing 1: Example topo

0: jdbc:hive2://xxx.com:10000> insert overwrite local directory '/xxx row format delimited fields terminated by '|' null defined as '' select * from xx.yy limit 20;Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xxx] does not have [WRITE] privilege on [/xxx] (state=42000,code=40000)Same query (INSERT OVERWRITE LOCAL DIRECTORY) works fine with Hive CLI:hive> insert overwrite local directory '/xxx' row format delimited fields terminated by '|' null defined as '' select * from xx.yy limit 20;Query ID = xx_abae1e37-0e40-4743-a7c6-a33ca9e5156cTotal jobs = 1Launching Job 1 out of 1Status: Running (Executing on YARN cluster with App id application_1492503032060_2596)--------------------------------------------------------------------------------VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED--------------------------------------------------------------------------------Map 1 .......... SUCCEEDED 4 4 0 0 0 0Reducer 2 .....

Often the namenode log files grow in size and they have too many kinds of messages. One of the most commonly faced scenario is where there had been multiple state changes in hdfs and investigating them becomes a pain when there are multiple occurrence in huge log files.Luckily there is a very easy way to make a few configuration changes to ensure that state change log statements get logged to a separate file.To isolate and log state change log messages to another file, add the following to hdfs-log4j and restart the namenodes. You can make this change from Ambari.Ambari > HDFS service > Configs tab > Advanced tab > Advanced hdfs-log4j section > hdfs-log4j template# StateChange log log4j.logger.org.apache.hadoop.hdfs.StateChange=INFO,SCL log4j.additivity.org.apache.hadoop.hdfs.StateChange=false log4j.appender.SCL=org.apache.log4j.RollingFileAppender log4j.appender.SCL.File=${hadoop.log.dir}/hdfs-state-change.log log4j.appender.SCL.MaxFileSize=256MB log4j.appender.SCL.MaxB

Hortonworks DataFlow (HDF) includes Apache Nifi with a wealth of processors that make the process of ingesting various syslogs from multiple servers easy. Information collected from the syslog can be stored on the HDFS distributed filesystem as well as forwarded to other systems as Spunk. Furthermore you can parse the stream and select which information should be stored on HDFS and which should be routed to an indexer on Splunk.To demonstrate this capability let us first review the Nifi ListenSyslog processor:The above processor corresponds to the syslog configuration in /etc/rsyslog.conf which includes the following line:...*.* @127.0.0.1:7780This will invoke syslog messages to be stream with Nifi flow which we can direct to another processor - PutSplunk, it was configured as follows:In the spunk UI you can configure data inputs under Setting->Data input -> TCP - Listen on a TCP port for incoming data, e.g. syslog.:To complete the selection use the port corresponding to the one

After Kerberos has been enabled, I was not able to open Hive View from Ambari.I would get the following error message:Issue detectedService 'userhome' check failed: Usernames not matched: name=root != expected=ambari-server-<clusterName>Service 'userhome' check failed:java.io.IOException: Usernames not matched: name=root != expected=ambari-server-<clusterName> at sun.reflect.GeneratedConstructorAccessor248.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) Root causeAmbari server is running as root so it tries to authenticate with a proxy user as 'root' where as the ambari.service.keytab expects a principal as ambari-server-<clusterName>@REALM. Hence the mismatch.Solution1. Edit the view settingsGo to Edit View page on Ambari :Manage Ambari > Views > Hive > Hive ViewOr simply:http://<ambariHost:port>/views/ADMIN_VIEW/<ambari.version>/INSTANCE/#/views/HIVE/versions/&l

PROBLEM: While creating an external Hive table using Hive Storage Handler for Phoenix, the table creation fails if schema name is added along with Phoenix table name. STEPS TO REPRODUCE: 1. Hive DDL for the table:create external table test.hive_e_phoenix1 (scd string,scddesc string)STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'TBLPROPERTIES ('phoenix.table.name'='myschema.mytbl','phoenix.zookeeper.quorum'='node1,node2,node3','phoenix.rowkeys'='scd');2. Following exceptions are seen.2016-07-17 16:02:25,898 ERROR [HiveServer2-Background-Pool: Thread-202]: operation.Operation (SQLOperation.java:run(209)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Phoenix table bigstats.bigstats_cd doesn't exist) at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:315) at org.apache.hive.service.cl

1.You can execute below api to get a list of hosts in your cluster in file hostcluster.txt # curl -s -u admin:admin http://ambari:8080/api/v1/hosts|grep host_name|sed -n 's/.*"host_name" : "\([^\"]*\)".*/\1/p'>hostcluster.txt2. In the loop you can write the api which need to run on the nodes :~~~while read line ;doj=$linemkdir -p $jdone < hostcluster.txt~~~ -admin:admin : username:password - Above loop will take each entries from file hostcluster.txt and need to execute the. 3. In order to install the clients you can use below API’s. Below API’s will install only 5 clients as mentioned below :####### Installing HDFS_CLIENT, YARN_CLIENT, ZOOKEEPER_CLIENT and MAPREDUCE2_CLIENT on "HOSTANAME" as following: +++++++++++ # curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d '{"RequestInfo":{"context":"Install HDFS Client"},"Body":{"host_components":[{"HostRoles":{"component_name":"HDFS_CLIENT"}}]}}' http://ambari-hostname:8080/api/v1/clusters/rest/hosts?Hosts/host_name=$j

PROBLEM: Balancer fails in few minutes without any block movement. SYMPTOMS: Following are the messages balancer exits with:- 16/11/22 07:08:29 DEBUG ipc.Client: IPC Client (280134559) connection to xxx.corp.example.com/0.0.0.0:8020 from hdfs-EST@HADOOP.XXX.CORP.EXAMPLE.COM got value #119316/11/22 07:08:29 DEBUG ipc.ProtobufRpcEngine: Call: getBlocks took 2486msNo block has been moved for 5 iterations. Exiting...Nov 22, 2016 7:08:29 AM 4 0 B 35.86 TB 200 GB ROOT CAUSE: The rack distribution looked like below:- /default-rack : 91 /Example1 : 18 /Example2 : 2 The 100% utilized nodes which we were trying to balance to create space were those 20 nodes registered with racks /Example1 and /Example2.Thus based on following rack awareness rules in balancer (rule#3 for this issue) for block placement, it was not at all possible for even a single block to move compromising fault tolerance. /** * Decide if the block is a good candidate to be

For a long time, when there was a big job to do, people relied on horses. Whether the job required heavy pulling, speed, or anything in between. However, not all horses were fit for every task. Certain breeds were valued for their incredible speed and endurance, especially when an important letter had to be delivered. Others were prized for their ability to carry and pull large payloads whether it was a fully armored knight or huge stone blocks. Today we rarely rely on horses and much more on technology to get many of the same kinds of jobs done. Very high volume streaming data is increasingly common in all lines of business because of the value and utility that it often carries. Horses will be of little help with this type of workload but luckily there is a whole host of tools to deal with streaming data. However, just like with horses, choosing the the right streaming tool for a particular use case is critical to the success of the project. Consider a use case where a direct

Apache NiFi allows us to rapidly create and operate very flexible and powerful dataflows. There are times, however, when the full flexibility and power of NiFi may not be required for the task at hand. For these times, MiNiFi may be a good fit. In particular, MiNiFI - C++ is worth considering when resources such as memory and compute power are constrained to such an extent that it is not feasible to run a full Java virtual machine.We are going to demonstrate how to deploy a MiNiFi - C++ dataflow to a cloud compute node that has only 64 megabytes of RAM. These types of nodes may be useful as a cost-savings measure, because cloud compute services typically charge based on resource usage.We'll start by cloning the latest nifi-minifi-cpp src:$ git clone https://github.com/apache/nifi-minifi-cpp.git$ cd nifi-minifi-cpp/Since this demo relies on a few commits which are not yet merged into master, we'll cherry pick the commits:$ git remote add achristianson https://github.com/achristianson/ni

Recently Atlas was enhanced to allow a UI friendly way of searching for entities present in atlas. The improvement has been introduced in two parts, one being the REST API ( /api/atlas/v2/search/basic ) and the other one is a rework of the search UI itself which adds a popup dialog for specifying the attributes of the type or the tag or both.Atlas Search details: http://atlas.apache.org/Search.htmlSearch REST API: http://atlas.apache.org/api/v2/resource_DiscoveryREST.html#resource_DiscoveryREST_searchWithParameters_POSTHere are some screenshots of the search experience.1. Landing Page2. Type-attribute based searchNotice how the drop down changes when a different type is selected.3. Tag-attribute based searchesEach attribute type (string, boolean, int, float, date etc.) has a certain set of operators that can be used on the attributes. These details can be found at the link present in the previous paragraph.Once the search results are displayed, there's an option to dynamically retrieve

Apache Atlas offers enterprises the capability to implement Data Governance on their HDP cluster. This is a game changer for enterprises that are looking to find ways to become more agile and effective in how they leverage data assets across the organization. Enterprises no longer have to see their data volumes grow and become unmanageable. With things like Asset Tagging, Data Lineage & Data Impact as well as Taxonomy, organizations can have a Business Catalogue at their fingertips allowing assets to be searched for and leveraged without any prior knowledge of the assets existence. In this Intro video we look at Asset Tagging as well as the Data Lineage and Data Impact functionality that tells you where data came from, how it was transformed along the way and what assets are affected downstream.

IBM DSX (Data Science Experience) Is a powerful tool that brings together Jupyter notebooks and RStudio into a single platform where Data Scientists can collaborate and share data assets and notebooks in an effective and streamlined way. In this Intro video we will cover some great aspects of DSX that everyone should know about when getting started and we will finish with the basics of accessing data assets that you have loaded into DSX. https://youtu.be/hb7o0NhV87k

Part 1: Installation and SetupNVIDIA Maxwell ™, 256 CUDA coresQuad ARM® A57/2 MB L24K x 2K 30 Hz Encode (HEVC) 4K x 2K 60 Hz Decode (10-Bit Support)4 GB 64 bit LPDDR4 25.6 GB/s2x DSI, 1x eDP 1.4 / DP 1.2 / HDMI16 GB eMMC, SDIO, SATAUp to 6 Cameras (2 Lane) CSI2 D-PHY 1.1 (1.5 Gbps/Lane)UART, SPI, I2C, I2S, GPIOs1 Gigabit Ethernet, 802.11ac WLAN, BluetoothPart 2: Classifying Images with ImageNetPart 3: Detecting Faces in ImagesPart 4: Using MiniFi to Send the Data and NiFi to Consume and ConvertBuild the Config Fileminifi-toolkit-0.2.0/bin/config.sh transform TensorRTMiniFi.xml config.ymlNote:Do not install minifi as a service, I had issues with this version of ubuntu and thatReferences:https://nifi.apache.org/minifi/system-admin-guide.html minifi.sh flowStatus processor:TailFile:health,stats,bulletins https://nifi.apache.org/minifi/getting-started.htmlhttps://unsplash.it/

This article seeks to walk you through the process developed in order to classify a given set of images into one of the x number of categories with the help of training datasets (of images) & a deep learning image recognition model "InceptionV3" & RanomForest classification algorithm. The technologies used are tensorflow & spark on hadoop platform. It inovles the following modules:Technology Stack:Spark 2.1.0, Python 3.5, HDP 2.6.1Prepare the Training datasets:1. Install the ImageMagick package using "yum install imagemagick"2. Obtain sample images from the customer and also the label that needs to be associated with them2. Extract the label (metadata) from each image using "ImageMagick" tool and place each image into different foldersprepareTrainingData.shThis is a bash script that extracts the metadata from the image and puts them into different bins say category 1 through 5. These images were manually classified by analyzing them with naked eye. These images will be us

Part 1: Installation and SetupPart 2: Classifying Images with ImageNetPart 3: Detecting Faces in ImagesNVidia also provides a good example C++ program for detecting faces, so we try that out next. You can add more training data to improve results, but it found me okay.In the next step we'll connect to MiniFi.Shell Source:root@tegra-ubuntu:/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin# ./facedetect.shdetectnet-console args (4): 0 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/detectnet-console] 1 [backupimages/tim.jpg] 2 [images/outputtim.png] 3 [facenet]detectNet -- loading detection network model from: -- prototxt networks/facenet-120/deploy.prototxt -- model networks/facenet-120/snapshot_iter_24000.caffemodel -- input_blob 'data' -- output_cvg 'coverage' -- output_bbox 'bboxes' -- threshold 0.500000 -- batch_s

Part 1: Installation and SetupPart 2: Classifying Images with ImageNetUse This Project: https://github.com/dusty-nv/jetson-inferenceThis will create a C++ executable to run Classification for ImageNet with TensorRT.Shell Call Exampleroot@tegra-ubuntu:/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin# ./runclassify.shimagenet-console args (3): 0 [/media/nvidia/96ed93f9-7c40-4999-85ba-3eb24262d0a5/jetson-inference-master/build/aarch64/bin/imagenet-console] 1 [backupimages/granny_smith_1.jpg] 2 [images/output_0.jpg]imageNet -- loading classification network model from: -- prototxt networks/googlenet.prototxt -- model networks/bvlc_googlenet.caffemodel -- class_labels networks/ilsvrc12_synset_words.txt -- input_blob 'data' -- output_blob 'prob' -- batch_size 2[GIE] attempting to open cache file networks/bvlc_googlenet.caffemodel.2.tensorcache[GIE] loading network profile from

Use Case:Ingesting sensors, images, voice and video from moving vehicles and running deep learning in the running vehicle. Transporting data and messages to remote data centers via Apache MiniFi and NiFi over Secure S2S HTTPS.Background:NVidia Jetson TX1 is a specialized developer kit for running a powerful GPU as an embedded device for robots, UAV and specialized platforms. I envision it's usage in field trucks for intermodal, utilities, telecommunications, delivery services, government and other industries with field vehicles.Installation and SetupYou will need a workstation running Ubuntu 16 with enough disk space and network access. This will be to download all the software and push it over a network to your NVidia Jetson TX1. You can download Ubuntu here (https://www.ubuntu.com/download/desktop). Fortunately I had a MiniPC with 4GB of RAM that reformatted with Ubuntu to be the host PC to build my Jetson. You cannot run this from a Mac or Windows machine.You will need a

Kerberos is a widely-used authentication system and is used throughout the Hadoop ecosystem, in particular, for strong authentication. The KDC, or key distribution center, is the name for the Kerberos server application that exposes the Authentication Service and Ticket Granting Service, as well as hosting the Kerberos principal database.Active Directory is used in many enterprises for Identity and Access Management. A common architecture in enterprise deployment scenarios is to make use of a local cluster KDC (often using the MIT-KDC packaging) to host the service and host principals associated with the cluster, and to configure a one-way trust between the AD domain and the cluster realm. This has the advantage of offloading Kerberos traffic from the domain controller(s) and not all enterprises do not necessarily want to host cluster principals within their AD domain. So how does a one-way trust work? The first thing to note is that the trust is instantiated by the existence of a spec

(I think) this occurred after upgrading from Ambari 2.4.2 to 2.5.0.3 (only looked for this when on Ambari 2.5.1.0)Here are the messages in SmartSense ViewFound these messages in /var/log/ambari-server/ambari-server.log when clicking on launch SmartSense View14 Jul 2017 11:14:05,523 ERROR [ambari-client-thread-467377] ServerProxy:162 - Failed to execute GET /recommendations. Reason: Connection refused14 Jul 2017 11:14:13,695 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /context. Reason: Connection refused14 Jul 2017 11:14:14,900 ERROR [ambari-client-thread-467507] ServerProxy:162 - Failed to execute GET /checkconfig. Reason: Connection refusedDiff'ing current hst-server.ini versus a previous (think it was from Ambari 2.4.2) yields# diff conf/hst-server.ini conf_13_04_17_04_17.save/2,4d1< enable.flex.subscription = false< smartsense.id=X-00000000-X-00000000< flex.subscription.id =6a4> smartsense.id=X-00000000-X-0000000027,28d24< url = http://

MiNiFi has http configuration Listener feature which can pull flow file configuration from the remote command control server, validate and apply the new flow configuration. The new file configuration should be a valid flow configuration yaml file.The way to configure the URL endpoint, pull interval and related security parameters.in minifi.properties nifi.configuration.listener.type=http nifi.configuration.listener.http.url=https://localhost:8080 nifi.configuration.listener.pull.interval=1 sec if you want to enable client certificate nifi.https.need.ClientAuth=true nifi.https.client.certificate=./conf/client.pem nifi.https.client.private.key=./conf/client.key nifi.https.client.pass.phrase=./conf/password nifi.https.client.ca.certificate=./conf/nifi-cert.pem

Apache NiFi 1.1.0 is now available and once again I want to test it in a isolated environment. Docker!The steps to do this are extremely similar to what has been detailed here (https://community.hortonworks.com/articles/69043/launching-a-nifi-docker-instance.html)Pull the imagedocker pull sunileman/nifi1.1.0You may find the mirror site is not optimal based on your location. Go here and grab your mirror site. Update MIRROR_SITE parameter in the Dockerfile with your mirror site. The Dockerfile is available here. If you update the Dockerfile you will have to build an image. Do this by runningdocker build --no-cache -t sunileman/nifi1.1.0 .Whether you pulled the image or built a new one, run this to launch Apache NiFi 1.1.0docker run -it --rm -p 8080-8081:8080-8081 sunileman/nifi1.1.0NiFi UI should be available herehttp://localhost:8080/nifi/Have fun!