Community Articles

This article describes various "recipes" on how to accomplish certain tasks with the NiFi processor ExecuteScript, with examples given in Groovy, Jython, Javascript (Nashorn), and JRuby. This is Part 2 in the series, I will be discussing reading from and writing to flow file contents, as well as error handling. Part 1 - Introduction to the NiFi API and FlowFiles Getting a flow file from an incoming queue Creating new flow files Working with flow file attributes Transferring a flow file Logging Part 2 - FlowFile I/O and Error Handling Reading from a flow file Writing to a flow file Reading and writing to/from a flow file Error Handling Part 3 - Advanced Features Using Dynamic Properties Adding Modules State Management Accessing Controller Services Introduction to FlowFile I/O Flow files in NiFi are made of two major components, attributes and content. Attributes are metadata about the content / flow file, and we saw how to manipulate them using ExecuteScript in Part 1 of this seri

If Kerberos for Hadoop cluster is implemented using enterprise AD, any Windows machine where users can sign-on using AD credentials has a cached ticket available. This cached ticket is available to any Windows applications by default, but any JAVA applications can't access cached ticket. To access cached ticket from a JAVA application, following registry entry in Windows should be setHKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\ParametersValueName: AllowTGTSessionKeyValueType: REG_DWORDValue:1Using klist command on Windows machine, verify username and REALM are in correct case as specified in kerberos settings on cluster. Then, create a jaas.conf file with following entryClient {com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=falseuseTicketCache=true;};To access kerborized cluster, JAVA program should be launched with following parameters.-Djava.security.auth.login.config="<path-to-jaas-conf>/jaas.conf"This will allow JAVA program to access cach

I had a hard time finding sample code online that uses the latest HBase 1.1+ Java APIs to connect to HDP 2.3 Sandbox. Here is my version for those that already know the Java basics.public static void main(String[] args) throws IOException { TableName tableName = TableName.valueOf("stock-prices"); Configuration conf = HBaseConfiguration.create(); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("hbase.zookeeper.quorum", "sandbox.hortonworks.com"); conf.set("zookeeper.znode.parent", "/hbase-unsecure"); Connection conn = ConnectionFactory.createConnection(conf); Admin admin = conn.getAdmin(); if (!admin.tableExists(tableName)) { admin.createTable(new HTableDescriptor(tableName).addFamily(new HColumnDescriptor("cf"))); } Table table = conn.getTable(tableName); Put p = new Put(Bytes.toBytes("AAPL10232015")); p.addColumn(Bytes.toBytes("cf"), Bytes.toBytes("close"), Bytes.toBytes(119)); table.put(p); Result r = table.get(new G

Short Description:Spark Hbase Connector (SHC) is currently hosted in Hortonworks repo and published as spark package. Below is simple example how to access Hbase table in Spark shell and Load the data into DataFrame. Once data is in Dataframe we can use SqlContext to run queries on the DataFrame.ArticleThe documentation here leaves out a few pieces in order access HBase tables using SHC with spark shell. Here is the Example accessing Hbase "emp" table in Spark shell. Hbase ShellCreate a simple "emp" Hbase table using Hbase shell and insert sample datacreate 'emp', 'personal data', 'professional data'put 'emp','1','personal data:name','raju'put 'emp','1','personal data:city','hyderabad'put 'emp','1','professional data:designation','manager'put 'emp','1','professional data:salary','50000'Once created exit Hbase shell and run spark shell providing SHC package and hbase-site.xml/usr/hdp/current/spark-client/bin/spark-shell --packages zhzhan:shc:0.0.11-1.6.1-s_2.10 --files /etc/hbase/conf

IntroductionThis wiki page describes the script sqoopTables.sh (GitHub: https://github.com/sourygnahtw/hadoopUtils/blob/master/scripts/sqoop/sqoopTables.sh).The purpose of the script is to sqoop in parallel many tables and to store them into Hive.MotivationsIn a previous project, I needed to download around 400 tables (out of 500) of a SQLserver database.Most of the tables were quite small (less than a few MB), which means that the overhead of fetching the metadata in Sqoop (establishing the connexions, getting the DDL of the table...) is very important compared to the time to do the real job (download the data). Getting the 400 tables could take around 6 hours.To speed up the downloading process, Sqoop usually "splits" a table in 4 parts, to parallelise the downloading process in 4 streams. In my project, such approach was not working: Many tables had not clear primary keys or the distribution of those keys was not uniform. Trying to find better split columns for 400 tables was a wast

Batch Loading with Druid and Hive IntegrationThis brief tutorial will walk through the various ways of ingesting raw data into druid and exposing the data in hive for analytics.For Reference, this tutorial was built with CentOS 7, HDP 3.0.1 w/ LLAP enabled.This tutorial will be divided into 6 sections:Data GenerationDefine the Ingest ScriptSubmit the Ingest SpecAccessing Data in HiveDouble the Data Double the FunLoading Data With HiveWhy Druid?Druid is a distributed columnar data store that is optimized for OLAP style queries. Data loaded into Druid is pre-aggregated based upon selected dimensions and metric fields. The data is further partitioned by time to enable efficient temporal access. This time based partitioning can be as long as a year and as small as a second, depending on what the business needs are.Today, Druid is tightly integrated with Hive which helps bring high speed OLAP analytics to the data lake. While Hive enhancements like LLAP deliver great performance improvement

Objective:Connect source (nifi) and destination (nifi) via site-to-site over SSL (https - 9091 port) Issue:NiFi Site to Site SSLHandShakeException PKIX path building failed - Missing TrustStore CertsRoot cause:Nifi truststores missing certificates of other Nifi Steps Taken:1. Run following command to get the list of Certificates in current Nifi TrustStore keytool -v -list -keystore <trustStoreLocation>2. If no certs that belong to target Nifi are found then we can proceed with installing the target Nifi cert3. Run following command to get the target Nifi public certificate echo -n|openssl s_client -connect <targetNiFiHostName>:9091 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/examplecert.crt3. Run following command to get the target Nifi public certificatekeytool -import -file /tmp/examplecert.crt -alias <targetNiFiCertificate> -keystore <trustStoreLocation> -storepass xxxx -noprompt 3. Verify using Openssl or SSLPoke commands java -Djavax.ne

IntroductionThis wiki page describes the script validationHiveTablesLinuxMode.sh (GitHub: https://github.com/sourygnahtw/hadoopUtils/blob/master/scripts/hive/validationHiveTablesLinuxMode.sh).The purpose of the script is to compare one Hive table with another, typically to validate a table generated by a Hive transformation against a source-of-truth (sqooped) table.MotivationsMy current project is the typical Hadoop project that consists of offloading a ETL process currently done in a relational database (SQLserver) to a Hadoop process.That involves rewriting all the (many) SQL queries in the ETL process to HiveQL queries.Some criteria of success for such project are:is the new Hive ETL workflow faster than the old one?have the output tables of the new workflow exactly the same data as in the original workflow?(does the customer pay less licenses and hardware with this new workflow?)So, getting sure that the HiveQL queries written in such ETL offloading project is a key subject. Hence

IntroductionLet's jump into tutorial 2 from my AI to Edge series!This tutorial details the creation of a Nifi flow executing the ONNX model we trained in my last article.More precisely we will try to feed these 3 handwritten digits and predict their value:Note: as always, all code/files referenced in this tutorial can be found on my github, here.AgendaBelow is an overview of the flow:As you can see, the flow is divided in the following sections:Section 1: Listening to a folder for new png filesSection 2: Resizing these images to 28x28 (size used to train our model)Section 3: Converting these images to CSV (format used to train our model)Section 4: Running our predictive modelSection 1: Listening to a folder for new png filesStep 1: Setup a variable for the root folderThis will be useful when we deploy the flow to a minifi flow. Go to your variables and create the following:Name: root_folderValue: location of your download of my githubStep 2: List files in folderCreate a ListFiles proce

Step 1 : Log into AWS your credentialsStep 2 : From the AWS console go to the following options and create a user in for the demo in AWSSecurity & Identity --> Identity and Access Management --> Users --> Create New UsersStep 3 : Make note of the credentialsawsAccessKeyId = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';awsSecretAccessKey = 'yyyyyyyyyyyyyyyyyyyyyyyyyyy';Step 4 : Add the User to the Admin Group by clicking the button “User Actions” and select the option Add Users to Group and add select your user (admin)Step 5 : Assign the Administration Access Policy to the User (admin)Step 6 : In the AWS Console , Go to S3 and create a bucket “s3hdptest” and pick your region Step 7 : Upload the file manually by using the upload button. In our example we are uploading the file S3HDPTEST.csvStep 8 : In the Hadoop Environment create the user with the same name as it is created in the S3 EnvironmentStep 9 : In Ambari do all the below properties in both hdfs-site.xml and hive-site.xml<pro

The Log Search service may use SSL for three things:Encrypt communication between Solr and the Log Search Server / Log FeedersEncrypt communication with the portal by using https protocolEncrypt communication with the Ambari Server for authenticationIn all of these cases the Log Search Server should have a trust store / key store available which should be configured in the logsearch-env.sh file. In case #1 the Log Feeders should also have a trust store / key store available which should be configured in logfeeder-env.sh.How to configure to use SSL in these cases?For all cases you should configure the stores for the Portal (and the Log Feeders in case of #1), see below how to do it.Using SSL for the communication with Solr you should configure your Solr server to use SSL.Using https as the portal protocol you should set the property Advanced logsearch-env / logsearch_ui_protocol to https. If you are not using Ambari you should add (or update the value of) logsearch.protocol=https to the

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. This article includes a Matrix that helps developers to take advantage of the latest NiFi processor to improve old and develop new flows. The following is a table of the matrix, arranged alphabetically from K-Z:See here for the Matrix Table from A-DSee here for the Matrix Table E-JNiFi 1.5NiFi 1.4NiFi 1.3NiFi 1.2ListAzureBlobStorageListAzureBlobStorageListAzureBlobStorageListAzureBlobStorageListDatabaseTablesListDatabaseTablesListDatabaseTablesListDatabaseTablesListenBeatsListenBeatsListenBeatsListenBeatsListenGRPCListenGRPCListenHTTPListenHTTPListenHTTPListenHTTPListenLumberjackListenLumberjackListenLumberjackListenLumberjackListenRELPListenRELPListenRELPListenRELPListenSMTPListenSMTPListenSMTPListenSMTPListenSyslogListenSyslogListenSyslogListenSyslogListenTCPListenTCPListenTCPListenTCPListenTCPRecordListenTCPRecordListenUDPListenUDPListenUDPListenUDPListenUDPRecordListenUDPRecor

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. This article includes a Matrix that helps developers to take advantage of the latest NiFi processor to improve old and develop new flows. The following is a table of the matrix, arranged alphabetically from E-J:See here for the Table Matrix from A-DSee here for the Table Matrix K-ZNiFi 1.5NiFi 1.4NiFi 1.3NiFi 1.2EncryptContentEncryptContentEncryptContentEncryptContentEnforceOrderEnforceOrderEnforceOrderEnforceOrderEvaluateJsonPathEvaluateJsonPathEvaluateJsonPathEvaluateJsonPathEvaluateXPathEvaluateXPathEvaluateXPathEvaluateXPathEvaluateXQueryEvaluateXQueryEvaluateXQueryEvaluateXQueryExecuteFlumeSinkExecuteFlumeSinkExecuteFlumeSinkExecuteFlumeSinkExecuteFlumeSourceExecuteFlumeSourceExecuteFlumeSourceExecuteFlumeSourceExecuteGroovyScriptExecuteProcessExecuteProcessExecuteProcessExecuteProcessExecuteScriptExecuteScriptExecuteScriptExecuteScriptExecuteSparkInteractiveExecuteSQLExecute

Building an Apache NiFi processor is super easy. I have seen/read several articles on how to get started by executing maven commands via CLI. This article is geared towards individuals who like to use an IDE (specially IntelliJ) to do the imports instead of running via CLI.On IntelliJ click on create project, Check "Create from archetype", click on "ADD Archetype" and enter the followingGroupId: org.apache.nifiArtifactId: nifi-processor-bundle-archetypeVersion: <YourVersionOfNifi>and then click "OK"Now your new NiFi Archetype has been created. Select itEnter GroupId, ArtifactId, and Version of your choiceA final attribute we need to add is artifactBaseName. This is mandatory. Click on "+" and enterName: artifactBaseNameValue: whatEverYouLikeNow a project ready to build a custom processor.Enjoy!

Following are the steps to connect to Phoenix tables using Spark2.1) Create a symlink of hbase-site.xml in spark2 confln -s /etc/hbase/conf/hbase-site.xml /etc/spark2/conf/hbase-site.xml2) Launch spark-shell using phoenix spark jars in extra classpath.spark-shell --conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar" --conf "spark.driver.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar"3) Create a phoenix connection and query the tables.scala> import org.apache.spark.sql.SQLContextimport org.apache.spark.sql.SQLContextscala> val sqlContext = new SQLContext(sc)sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@495e8a3scala> val df = sqlContext.load("org.apache.phoenix.spark",Map("table" -> "TABLE1", "zkUrl" -> "localhost:2181"))df: org.apache.spark.sql.DataFr

UPDATE: This article has been vetted against HDF 2.0 - HDF 3.2. Minor updates have been made for additional clarity on use of NiFi CA for establishing trust with Ranger. Prerequisites NiFi has been installed running with SSL enabled (with certificates manually installed or using the NiFi Certificate Authority). You will need the keystore/truststore names, locations, aliases, identity (DN) and passwords used when enabling SSL for NiFi. Ensure that all nodes have the same keystore/truststore passwords applied and like named locations in order to apply ranger nifi plugin configurations consistently via Ambari. Ranger has been installed and configured with security enabled. For instructions on setting up SSL for Ranger please review Configure Ambari Ranger SSL . Note the name, location, aliases, identity (DN) and passwords used when creating keystores and truststores for Ranger If Kerberos will be used recommend that this is enabled for the HDF cluster before proceeding. Part 1 - Estab

To enable HTTPS for web HDFS, do the following: Step 1: Get the keystore to use in HDFS configurations.a) In case cert is getting signed by CA, do the following:1. Generate a keystore for each host. Make sure the common name portion of the certificate matches the hostname where the certificate will be deployed.keytool -genkey -keyalg RSA -alias c6401 -keystore /tmp/keystore.jks -storepass bigdata -validity 360 -keysize 20482. Generate CSR from above keystorekeytool -certreq -alias c6401 -keyalg RSA -file /tmp/c6401.csr -keystore /tmp/keystore.jks -storepass bigdata3. Now get the singed cert from CA - file name is /tmp/c6401.crt4. Import the root cert to JKS first. (Ignore if it already present)keytool -import -alias root -file /tmp/ca.crt -keystore /tmp/keystore.jksNote: here ca.crt is root cert5. Repeat step4 for intermediate cert if there is any.6. Import signed cert into JKSkeytool -import -alias c6401 -file /tmp/c6401.crt -keystore /tmp/keystore.jks -storepass bigdata7. I

IntroductionBefore the official release of Cloudera Data Flow (ex. Hortonworks Data Flow), you may want to play with Nifi and Hive on CDH.However, because CDH 5 is using a fork of Hive 1.1, the HiveQL processors and controller services included on the official Apache release will not work, so you need to have your own, as explained in this article: Connecting NiFi to CDH Hive.This article is awesome, but does not focus on Kerberos/SSL; since I had to do the configuration myself, I thought I would share the knowledge.Note: You could use a DBCP connection to connect to Cloudera Hive but it will not allow you to use the proper authentication.Pre-RequisitesTo connect to Hive with SSL and Kerberos, you will need the following:A running version of Nifi (I used Apache 1.9 in this example)A Kerberized CDH cluster with Hive on SSL (I used CDH 5.15 in this example)The certificate to add to your keystore for SSL connectionA keytab for a specific user authorized in the clusterThe krb5 configuratio

In the previous article, we saw how to stream tweets using NiFi, Kafka, Tranquility, Druid and Superset ...https://community.hortonworks.com/articles/177561/streaming-tweets-with-nifi-kafka-tranquility-druid.html You have to implement in that previous article, the part of Druid datasource and Nifi flow to carry on here. But life already is hard enough, why not simplify it? The idea here is to perform the same streaming but now integrating Nifi directly to Druid. So, our new diagram would look like this: As we saw in that article, we have Tranquility as an integrating factor between Kafka and Druid. Some people asked me: Why not use Kafka Indexing Service instead of Tranquility? My answer: because Tranquility as a framework, can be used flexibly, doing integration of almost any component to the Druid. Thus, on this nifi-druid integration, we will build a custom Nifi processor, which uses Tranquility to integrate data directly into Druid. Ok, it’s time to hands on! Let

In NiFi/HDF, it is possible to create a kind of lookup table of key/value pairs using a DistributedMapCacheServer. The DistributedMapCacheServer is used by various processors such as GetHBase and DetectDuplicate. It can also be leveraged by users in their flows, with the PutDistributedMapCache and FetchDistributedMapCache processors, by specifying a corresponding DistributedMapCacheClientService.Sometimes, however, it might be the case that the user would like to interact with the DistributedMapCacheServer programmatically (and external to NiFi), say for removing specific entries, inserting/populating entries, etc. To that end I have written a Groovy script (dcachegroovy.txt, rename to dcache.groovy) to allow manipulation of the entries in a DistributedMapCacheServer from the command-line, e.g.The usage is as follows:Usage: groovy dcache.groovy <hostname> <port> <command> <args> Where <command> is one of the following: get: Retrieves the values for the key

IntroductionTime for the tutorial 1 of a series detailing how to go from AI to Edge!Note: all code/files referenced in this tutorial can be found on my github, here.AgendaThis tutorial is divided in the following sections:Section 1: Create a custom Docker container running Jupyter for CDSWSection 2: Automate Jupyter launch in a CDSW projectSection 3: Train and save a model reading MNSIT databaseSection 1: Create a custom Docker container running Jupyter for CDSWThis is fairly straight forward to implement, as it is detailed in the official documentation.Note: make sure that dock is signed in with your Dockerhub username/password (not email) otherwise the docker push will not work.Step 1: Create a repository in docker hubGo to docker hub and sign in with your account. Create a new repository as follows:You should see something like this:Step 2: Creating a custom docker fileGo to a folder on your computer can create this docker file (saving it as Dockerfile😞FROM docker.repository.cloude

IntroductionMachine Learning and Artificial Intelligence frameworks are numerous and their impact on the future of computer science need no emphasis. However, deploying these models can be complex and fairly manual without the appropriate multi-function ecosystem, especially when deploying these models to the edge.Luckily, Cloudera comprehensive data management suite make this endeavor very easy.During this series, I will present how to create a Deep Learning model trained to read digits from the MNIST database and deploy it to the edge.This article is an introduction to the architecture and pre-requisites necessary for this tutorial. It will refer to sub articles that will be tutorials that anyone can follow to implement understand how to take an AI model and operationalize it to the edgeArchitecture overviewThe figure below gives a highlight of my hybrid cloud platform:As you can see, it is comprised of the three main functions:Cloudera Data Science Workbench: data science hub used t

Introduction Parquet is a famous file format used with several tools such as Spark. NiFi can be used to easily convert data from different formats such as Avro, CSV or JSON to Parquet. This article explains how to convert data from JSON to Parquet using the PutParquet processor. Implementation Define a schema for the source data In this article I'll be using JSON data source with the following structure: {"created_at" : "Tue Oct 10 21:47:12 CEST 2017","id_store" : 4,"event_type" : "store capacity","id_transaction" : "6594277248900858122","id_product" : 319,"value_product" : 1507664832649} Since we will be using a record based processor, we need to define a schema for our data. This will be an Avro schema but it can be used with other types as well. It's only "a common langage" that helps us describe a schema. The Avro Schema for my data is the following: { "namespace": "nifi", "name": "store_event", "type": "record", "fields": [ { "name": "created_at", "type": "string" }, { "name": "

1. Goal Security is one of fundamental features for enterprise adoption. Specifically, for SQL users, row/column-level access control is important. However, when a cluster is used as a data warehouse accessed by various user groups via different ways like Apache Spark™ 1.6/2.1 and Apache Hive, it is difficult to guarantee access control in a consistent way. In this article, we show the use of Apache Ranger™ to manage access control policy for Spark SQL. This enhancement is done by integrating Apache Ranger and Apache Spark via `spark-llap`. We also show how to provide finer grained access control(row/column level filtering and column masking) to Apache Spark.2. Key Features Shared Policies: The data in a cluster can be shared securely and consistently controlled by the shared access rules between Apache Spark and Apache Hive. Audits: All security activities can be monitored and searched in a single place, i.e., Apache Ranger. Resources management: Each user can use different queues w

Objective This tutorial walks you through a NiFI flow that utilizes the ConvertRecord processor and Record Reader/Writer controller services to easily convert a CVS file into JSON format. Additionally, the flow is modified to also convert the CSV file to Avro and XML formats. Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0Convert CSV to JSONSupport Files Here is a template of the flow discussed in this tutorial: convert-csv-to-json.xml Here is the CSV file used in the flow: users.txt (Change the extension from .txt to .csv after downloading) Note: The CSV data originated from the site, RandomUser. This useful site provides a free API to pull down randomly generated user data. For example: https://randomuser.me/api/0.6/?results=10&forma