Community Articles

Creating NIFI workflowUse below steps to create the whole workflow.1. Open the Nifi Web UI, download the below processors and configure as described.a. MergeContentb. InferAvroSchemac. PutHive3streaming2. Now let’s configure them one by one. Right click each processor and configure as shown below.a. MergeContentb. InferAvroSchema – In the CSV header definition put all the headers of your data separated by delimiter.c. PutHive3Streaming – Here we need to configure the record reader. Right click on the processor > configure > Record reader > create new service.d. Add CSV reader and save.e. Click on the record reader again and then enable by clicking on middle sign on upper right-side corner.f. Connect all three processor and address the errors if any. For any configuration reference, you can try importing the attached workflow.I have attached my workflow in case you want to directly import into your Nifi.Rasp_new.xmlLinks to seriesPart 1, Part 2, Part 3, Part 4, Part 5

NIFI Installation on HDPWe will go through step by step instructions how to setup your own data pipeline. First to save time and resources we will install NiFi on HDP cluster using mpack.1. Install and setup the HDP cluster.2. Install the HDF mpack (versions may vary) on ambari server node and restart the ambari server.1. wget http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.2.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.2.0.0-520.tar.gz2. cp -r /var/lib/ambari-server/resources /var/lib/ambari-server/resources.backup3. ambari-server install-mpack --mpack=hdf-ambari-mpack-3.2.0.0-520.tar.gz  --verbose4. ambari-server restart3. Once we restart the ambari-server, we will be now able to add Nifi service to the same HDP cluster. Two different clusters can be used for HDP and HDF, however we are using only one to reduce the complexity.4. Select NiFi service5. Resume the installation after typing in the Master key, encryption and Nifi CA token password6. Once it is done, valid

INTRODUCTIONFrom Edge to AI: This tutorial is designed to help you walk through the process of creating a work flow to read data from a edge sensors (Raspberry Pi 3 here) and ingest in Hive through NiFi workflow. The MiNiFi flow will push data to a remote NiFi instance which then using Hive streaming features will ingest data to Hive tables. Once ingested, it can be analyzed with hive sql interface, Spark, zeppelin and by numerous other tools designed for Hadoop platform.In the whole process we will be going through the installation and configuration of below working components.NiFi - Apache NiFi commonly called Niagara Files, is an integrated data logistics platform for automating the movement of data between disparate systems. It provides real-time control that makes it easy to manage the movement of data between any source and any destination. It is data source agnostic, supporting disparate and distributed sources of differing formats, schemas, protocols, speeds and sizes such as m

Steps on how to setup YARN to run docker containers can be found in Part 1: articleIn this article I will show how to run Hive components (Hiverserver2, Metastore) as docker containers in YARN.Metastore will be using a Mysql 8 database also running as a docker container in a local host.Pre-requisites:1. Pull mysql-server image from Docker hub, run image as a docker container, create hive database and permissions for hive user :docker pull mysql/mysql-serverdocker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=admin  --restart=always --name mysqld mysql/mysql-serverdocker exec -it mysqld bashbash-4.2# mysql -u root --password=adminmysql> CREATE DATABASE hive;mysql> CREATE USER 'hive' IDENTIFIED BY 'hive';mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' WITH GRANT OPTION;2. Create user 'hive' and assign to 'hadoop' group:useradd hiveusermod -aG hadoop hiveDockerize Hive:1. Create a yum repo file "hdp.repo" that contains HDP-3.1.0.0 and HDP-UTILS-1.1.0.22 repositories:[HDP-3.

This article talks about two things:1. How to develop a simple python script which returns the result_code like 'CRITICAL', 'WARN', 'OK', 'UNKNOWN' based on certain conditions matching our requirement.2. How to write an alert definition in json format so that we can install those alerts to the Ambari Server in order to get the alerts.The scripts/files attached to this article are "test_alert_disk_space.py" and "alerts.json" that we are going to use as part of the following steps:Step-1). Create a python script as attached "test_alert_disk_space.py" that finds the list of mount points and the it returns the disk usage status of those mount points. The based on the percentages specified it returns the return code as 'CRITICAL', 'WARN', 'OK', 'UNKNOWN'Step-2). Place the file "test_alert_disk_space.py" in the following Path on the ambari-server : "/var/lib/ambari-server/resources/host_scripts/" Example:cp -f test_alert_disk_space.py /var/lib/ambari-server/resources/host_scriptsStep-3).

AbstractThis article objective is to cover key design and deployment considerations for High Availability Apache Kafka service.IntroductionKafka’s through its distributed design allows a large number of permanent or ad-hoc consumers, being highly available and resilient to node failures, also supporting automatic recovery. These characteristics make Kafka an ideal fit for communication and integration between components of large scale data systems. Kafka application resilience is not enough to have a true HA system. Consumers and producers or the network design need also to be HA. Zookeepers and brokers need to be able to communicate among themselves, producers and consumers need to be able to access Kafka API. A fast car can reach the maximum speed only on a good track. A swamp is a swamp for a fast or a slow car.Kafka DesignKafka utilizes Zookeeper for storing metadata information about the brokers, topics, and partitions. Writes to Zookeeper are only performed on changes to the memb

Joining Collections in SOLR (Part 1) Sometimes you may want to inner join data from one solr connection to another. There is a facility to perform this action using a join query in SOLR. The easiest way to perform the join is by linking a single attribute from one collection to another attribute in another collection. This join works very well for standalone indexes, but does not work well for distributed indexes. To do this in a distributed index, we’ll perform that in part II of this article. To demonstrate, let’s say we have two collections. Sales, which contains the amount of sales by region. And in the other collection called People, which has people categorized by their region and a flag if they are a manager. Let’s say our goal is to find all of the sales by manager. To do this, we will join the collections using region as our join key, and also filter the people data by if they are a manager or not. Here is the filter query (fq) in solr on how to make this happen: fq={!join

This article describes the setup of two separate KDCs in a Master/Slave configuration. This setup will allow two clusters to share a single Kerberos realm, which allows the principals to be recognized between clusters. A use case for this configuration is when a Disaster Recovery cluster is used as a warm standby. The high level information for the article was found at https://web.mit.edu/kerberos/krb5-1.13/doc/admin/install_kdc.html, while the details were worked out through sweat and tears.Execute the following command to install the Master and Slave KDC if the KDC is not already installed: yum install krb5-server The following defines the KDC configuration for both clusters. This file, /etc/krb5.conf, must be copied to each node in the cluster.[libdefaults] renew_lifetime = 7d forwardable = true default_realm = CUSTOMER.HDP ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false udp_preference_limit=1[domain_realm] customer.com = CUSTOMER.HDP .customer.com = C

  Short Description: This article targets to describe and demonstrate Apache Hive Warehouse Connector which is a newer generation to read and write data between Apache Spark and Apache Hive. 1. Motivation Apache Spark and Apache Hive integration has always been an important use case and continues to be so. Both provide their own efficient ways to process data by the use of SQL, and is used for data stored in distributed file systems. Both provide compatibilities for each other. As both systems evolve, it is critical to find a solution that provides the best of both worlds for data processing needs. In case of Apache Spark, it provides a basic Hive compatibility. It allows an access to tables in Apache Hive and some basic use cases can be achieved by this. However, not all the modern features from Apache Hive are supported, for instance, ACID table in Apache Hive, Ranger integration, Live Long And Process (LLAP), etc. as of Spark 2. Apache Spark supports a pluggable approach for va

HDP Search provides the tools to index data from your HDP cluster to Solr. You can utilize the power of connectors that are shipped with HDPSearch to index data from HDFS, Hive tables, and Spark dataframes to Solr. Once you have your data in Solr, search and querying is simpler. You may find the official Hortonworks documentation for HDP Search 4.0 here: https://docs.hortonworks.com/HDPDocuments/HDPS/HDPS-4.0.0/bk_solr-search-installation/content/ch_hdp-search.html In this article, you may find how you could set up your HDP Cluster with HDP Search using the Solr Management pack shipped by Hortonworks. Additionally, you will find details on how to use connectors for Hive, Spark, and Hdfs to index data to Solr and query from Solr. (This document assumes that you already have an Ambari-2.7.0 + HDP-3.0 cluster up and running) Setup Install Management Pack: Download Solr service mpack on Ambari server node: wget http://public-repo-1.hortonworks.com/HDP-SOLR/hdp-solr-ambari-mp/solr-service

If you would like to upgrade Cloudbreak in an environment with no internet access, refer to the following documentation:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/upgrade/content/cb_upgrade-cloudbreak-no-internet.html

If you would like to configure Cloudbreak to use an external database that uses SSL, refer to the following documentation:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/configure/content/cb_configure-external-ssl.html

Due to AMBARI-24283 BUG some times there would be warnings like below,2018-08-09 09:38:27,666 WARN - You have config groups present in the database with no corresponding service found, [(ConfigGroup, Service) => ( HDFS_DL111, null ), ( YARN_2222, null )]. Run --auto-fix-database to fix this automatically. Possible root cause: In some of the Ambari 2.5.x/2.6.x versions, service DELETE does not delete config groups cleanly and it shows the WARNING while starting ambari-server.To fix above WARNINGS below steps can be followed.1. Take Ambari database backup.2. Run below queriesdelete from confgroupclusterconfigmapping where config_group_id in (select group_id from configgroup where group_name in ( 'HDFS_DL111', 'YARN_2222'));delete from configgrouphostmapping where config_group_id in (select group_id from configgroup where group_name in ( 'HDFS_DL111', 'YARN_2222'));delete from configgroup where group_name in ( 'HDFS_DL111', 'YARN_2222');

ISSUE: Sqooping data from hdfs to mysql db works when we run it from command line.sqoop import --connect jdbc:mysql://mysqlserver.somedomain.com:3306/sample --username user1 --password password --table sample_test --target-dir /user/user1/data When we attempt to run the same in an oozie workflow the job fails with the following error;Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] SOLUTION: Setting the path to the oozie.libpath in the job.properties and copying the database connector jars to this directory resolved this issue. Original job.properties:oozie.use.system.libpath=Truesecurity_enabled=TruenameNode=hdfs://HD0credentials={u'hcat': {'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://server.somedomain.com:9083'), ('hcat.metastore.principal', u'hive/server.somedomain.com@SOMEREALM.COM')]}, u'hive2': {'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://server.somedomain.com:10000/default'), ('hive

NiFi has previously supported the ability to refer to flow file attributes, system properties and environment properties within expression language (EL); however the community requested an enhancement to also support custom properties. This would give users even more flexibility either in processing, handling flow content, or even in flow configuration (e.g. referring to a custom property in EL for connection, server or service properties). In NiFi versions 0.7 & 1.0.0 an enhancement was added to allow administrators to define custom property files on nodes within their cluster and configure NiFi with their location so those properties could be loaded and available within EL. A new field in the nifi.properties file ( nifi.variable.registry.properties) is available for an administrator to set the paths of one or more custom properties files for use by NiFi. Figure 1 - Custom Properties reference in nifi.properties Once the nifi.properties file is updated custom attribute

IMPORTANT:DO NOT FORGET to backup your Ambari database before executing DELETE API calls. This is tested on Ambari version 2.6.x & 2.7.1. Kindly use these steps on your Test cluster first OR Contact Cloudera Support for more details.Step 1: Take a backup of your Ambari database# mkdir /var/tmp/postgres_backup # pg_dump -U ambari ambari > /var/tmp/postgres_backup/$(date +"%Y%m%d%H%M%S")_ambari-bkp.sqlStep 2: Install jq package (If not available)# rpm -qa | grep jq# yum whatprovides jq# yum install jq -yStep 3: Get All (AD/LDAP) Users & Groups# curl --insecure -u admin:admin -H 'X-Requested-By: ambari' -X GET http://$(hostname -f):8080/api/v1/users?Users/ldap_user=true | jq -r '.items[].Users.user_name' > ambari-ldap-users.txt# curl --insecure -u admin:admin -H 'X-Requested-By: ambari' -X GET http://$(hostname -f):8080/api/v1/groups?Groups/ldap_group=true | jq -r '.items[].Groups.group_name' > ambari-ldap-groups.txtStep 4: Verify Users & Groups that needs to

Pre-requisites:1. Install Docker in all nodemanager hosts and configure:It is recommended to install the version of Docker that is provided by your Operating System vendor. The Docker package has been known by several names; docker-engine, docker, and docker-ce.yum install dockerIf having issues installing docker, following document can be followed:https://docs.docker.com/install/Edit ‘/etc/docker/daemon.json’ and add the following options:{        "live-restore" : true,        "debug" : true,        "dns": ["<YARN registry dns ip addr>"] }If not using HTTPS, Configure each of the cluster hosts to skip HTTPS checks by adding following line in ‘/etc/docker/daemon.json’"insecure-registries": ["<docker registry server>:5000"]2. Create a private local docker registry(Optional)2.1- Designate a server in the cluster for use by the Docker registry. Minimal resources are required, but sufficient disk space i

Pre-requisite:1. Setup Ambari with LDAP and Sync. 2. Setup Knox and point to same LDAP as Ambari server.Enable SSO for Ambari:1. Get the Knox public cert by running below.openssl s_client -connect KNOXHOST:8443 <<<'' | openssl x509 -out /tmp/knox.crt2. Run "ambari-server setup-sso"3. "provider URL": Enter https://<hostname>:8443/gateway/knoxsso/api/v1/websso4. "Public Certificate pem" : Provide step1 cert file content without BEGIN/END blocks. -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- Note: Make sure your /etc/ambari-server/conf/jwt-cert.pem file should have only one BEGIN/END5. You can select default for rest of the configs.6. Re-start Ambari server:Knox Configurations1. If Ambari and Knox is in different host then Whitelist Ambari URL. In Advanced knoxsso-topology modify below config for whitelisting all (or you can write regex for specific) <param> <name>knoxsso.redirect.whitelist.regex</name> <value

Using Raspberry Pi 3B+ with Apache NiFi MiNiFi and Google Coral Accelerator and Pimoroni Inky PhatFirst we need to unbox our new goodies. The Inky Phat is an awesome E-Ink display with low power usage that stays displayed after shutdown! Next I added a new Google Coral Edge TPU ML Accelerator USB Coprocessor to a new Raspberry Pi 3B+. This was so easy to integrate and get up and running.Let's unbox this beautiful device (but be careful when it runs it can get really hot and there is a warning in the instructions). So I run this on top of an aluminum case and with a big fan on it. Pimoroni Inky PhatIt is pretty easy to set this up and it provides a robust Python library to write to our E-Ink display. You can see an example screen here.https://github.com/pimoroni/inkyPimoroni Inky pHAT ePaper eInk Display in RedPimoroni Inky Phat (Red)https://shop.pimoroni.com/products/inky-phathttps://github.com/pimoroni/inkyhttps://pillow.readthedocs.io/en/stable/reference/ImageDraw.htmlh

Error:Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 configs from ZooKeeper (state=08S01,code=0)Issue: Beeline connection failing using Kerberos principal and ServiceDiscoveryMode beeline -u "jdbc:hive2://zk1.com:2181,zk2.com:2181,zk3.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=KERBEROS;principal=myuser/hiverserver2@EXAMPLE.COM;httpPath=cliservice;saslQop=auth-conf" Resolution: We do not need to pass the kerberos principal within the beeline command because we are using zookeeper. You must already have valid Kerberos ticket.Example:kinit myuserbeeline -u "jdbc:hive2://zk1.com:2181,zk2.com:2181,zk3.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf" Note: You must have a valid ticket first. If no valid ticket is present you will see an error like the following:The error in the log "echanism level: Fail

Ambari is the heart of any HDP cluster. It provides us the feature of provisioning, managing, monitoring and securing Hadoop / HDP clusters. It's is a Java program which interacts with Database to read the cluster details and runs on embedded jetty server. Many times we find issues with Ambari server performance. It's Ambari UI operations sometimes responds slowly or the startup might take longer time if it is not properly tuned. So in order to troubleshoot the ambari server performance related issues we should look at some of the data/stats and tuning parameters to make the ambari server perform better. In this article we will talk about some very basic tuning parameters and performance related troubleshooting. What all information's needed? When we notice that the ambari server is responding slow then we should look first the following details first: 1). The number of hosts added to the ambari cluster. So that accordingly we can tune the ambari agent thread pools. 2). The number of c

Hi,If you don't want SMARTSENSE in your cluster but still it comes as default selected component during install wizard then go through below steps to save yourself some trouble.Tried on HDP Version : 3.0 and 3.1Goto below path on the ambari-server node:/var/lib/ambari-server/resources/stacks/HDP/3.0/services/SMARTSENSE/metainfo.xmlOpen the above file in editor mode (e.g. vi)Uncomment or delete the below line [Line 23 may vary in different release]<selection>MANDATORY</selection>After making the above change restart ambari-server and proceed with cluster install wizard.Now SMARTSENSE won't be a mandate component.Thanks for reading.

https://youtu.be/-HMyEpDJeGgConfiguring Ambari 2.4.2 and HDP 2.5 for Kerberos using AD as the KDC Add bonus coverage of adding a new datanode to a HDP cluster that is secured. There are empty OUs created in AD to store hadoop principals/hadoop nodes (HadoopServices)Hadoopadmin user has administrative credentials with delegated control of "Create, delete, and manage user accounts" on above OUDelegate OU permissions to hadoopadmin for OU=HadoopServices. In 'Active Directory Users and Computers' app:right click HadoopServicesDelegate ControlNextAddhadoopadminchecknamesOKSelect "Create, delete, and manage user accounts"OKKDC:KDC host: ad01.prod.hortonworks.netRealm name: PROD.HORTONWORKS.NETLDAP url: ldaps://ad01.prod.hortonworks.netContainer DN: OU=HadoopServices,DC=prod,DC=hortonworks,DC=netDomains: prod.hortonworks.netKadmin:Kadmin host: ad01.prod.hortonworks.netAdmin principal: hadoopadmin@PROD.HORTONWORKS.NETAdmin password: xxxxxx

I was recently tinkering with the walmart rest-api. This is publicly available interface and can be used for a quick price look up for products. The overall goal of the project is to keep track of the cost of specific shopping carts day to day but this intermediate step provides an interesting example-case.The requirements of this stage:Use UPC codes to provide a lookupAvoid having the pass the walmart API key to the internal client making the callExtract features such as price in preparation for a JDBC database entry.Core Concepts in NifiNifi has the ability to serve as a custom restful api which is managed with the "HandleHttpRequest" and "HandleHttpResponce" processors. This can be a Get/Post or any of the other common typesNifi can make calls to an external rest API via the "InvokeHTTP" ProcessorXML Data can be extracted with the "EvaluateXPath" ProcessorThe HandleHttpRequest ProcessorThis processor receives the incoming rest call and makes a flow file with information pertainin

OverviewIn this article I show how to very quickly build custom logging against source log files. Adding new custom logging to the flow takes only seconds: each new custom log is copy-pasted from an existing one and 3 configuration properties are changed. The result is a single flow with many diverse custom logs against your system.Use CasesUse cases include:Application development: permanent or throway logging focused on debugging software issues or NiFi flows.Production: logging for focused metrics gathering, audting, faster troubleshooting, alertingAll envts: prefiltering Splunk ingests to save on cost and filtering out unneeded log dataUse your imagingation … how can this flow pattern make your life better / more effective?Flow DesignSingle customized logA single customized log is achieved by tailing a log file, splitting the lines of text, extracting lines that match a regex, specifiying a custom log filename for the regex-extracted lines, and appending the lines to that custom l