Resources

MiNiFi (Java Version) is essentially NiFi with a few differences and hence why it runs so darn well on containers/Kubernetes. The use case is to have a single management console (Edge Flow Manager) to manage 0 + many MiNiFi agents which require autoscaling on Kubernetes based on some arbitrary metrics...for example CPU/RAM threshold. EFM and NiFi Registry are required but don't need autoscaling; therefore, these services will be deployed on Azure Container Service. MiNiFi on the other hand often benefits from autoscaling and hence it will be deployed on Azure Kubernetes Service.Required for this demonstrationAzure subscriptionContainer RegistryDemo will leverage Azure Container RegistryKubernetes ServiceDemo will leverage Azure Kubernetes ServiceAzure CLIThe following images need to be stored in Azure Container RegistryEdge Flow Managerhttps://github.com/sunileman/efm1.0.0.0-dockerNiFi Registryhttps://github.com/sunileman/NiFi-Registry-ServiceMiNiFi (Java)https://github.com/sunileman/C

Steps to fix below Database consistency WARNINGs in Ambari 2.7.xWARN - You have config(s): webhcat-site-version1540591929056,webhcat-log4j-version1540591928580,hive-exec-log4j-version1540591929168,webhcat-env-version1540591929289,hive-log4j-version1540591929465,hcat-env-version1540591928892 that is(are) not mapped (in serviceconfigmapping table) to any service!CREATE TEMPORARY TABLE orphaned_configs AS(select config_id,type_name,version_tag from clusterconfig where unmapped != 1 and type_name not in ('cluster-env') and config_id not in (SELECTcc.config_id FROM clusterconfig cc, serviceconfig sc, serviceconfigmapping scmWHERE cc.type_name != 'cluster-env'AND cc.config_id = scm.config_idAND scm.service_config_id = sc.service_config_id));Update clusterconfig set unmapped = 1 where config_id in (select config_id from orphaned_configs);drop table orphaned_configs;If it is 2.6.x then following query can be usedSELECT cc.config_id, cc.type_name, cc.version_tagFROM ambari.clusterconfig cc, amb

I'm going to cover a simple OSS solution to Shredding XML in NiFi, and demonstrate how you can chain simple steps together to achieve common data shredding tasks. Feel free to get in touch if you need to achieve something more complex than these basic steps will allow. We will be covering: Procedurally converting Xml to Json using a fast XSLT 2.0 template Constructing Jolt transforms to extract nested subsections of JSON documents Constructing JsonPath expressions to split multi-record JSON documents Procedurally flattening complex nested JSON for easy querying This process is shown on NiFi-1.2.0, and tested on a variety of XML documents, but most notably a broad collection of GuideWire sample XMLs as part of a Client PoC. The XML examples below have retained the nested structure but anonymised the fields. XML to JSON Here we combine the NiFI TransformXML processor with the excellent BSD-licensed xsltjson procedural converter found at https://github.com/bramstein/xsltjson. Simply chec

LDAP security optionsLDAP over SSLTraditionally, LDAP connection is wrapped with SSL/TLS, and via a separate service port, normally 636. The protocol is ldaps://. To be aware, it’s not just over SSL, depends on configuration, can over TLS as well.LDAP with StartTLSAlternatively, StartTLS extended operation enable encrypted connection after the initial unencrypted connection, via the same plain ldap port, normally 389. If you don’t configure properly, it could be over SSL, not always with TLS. This protocol is new, normally is used to upgrade the plain ldap port 389 to be secure. But not all customer have configured it in AD/LDAP server. Most common cases are still use ldaps for secure ldap connection. So this article is focusing on LDAPS.Components support LDAPSMost of HDP/HDF components support LDAPS. Ambari, NiFi, Ranger Usersync, Ranger Admin, Zeppelin, Knox. But Atlas still only supports LDAP.General Configure LDAPS Steps Verify LDAPS certificateTo connect to ldaps service, first t

Hortonworks Data Flow 2.1 was recently released and includes a new feature which can be used to connect to an Azure Data Lake Store. This is a fantastic use case for HDF as the data movement engine supporting a connected data plane architecture spanning on-premise and cloud deployments.This how-to will assume that you have created an Azure Data Lake Store account and that you have remote access to an HD Insights head node in order to retrieve some dependent JARs. We will make use of the new Additional Classpath Resources feature for the GetHdfs and PutHdfs processors in NiFi 1.1, included within HDF 2.1. The following additional dependencies are required for ADLS connectivity: adls2-oauth2-token-provider-1.0.jar azure-data-lake-store-sdk-2.0.4-SNAPSHOT.jar hadoop-azure-datalake-2.0.0-SNAPSHOT.jar jackson-core-2.2.3.jar okhttp-2.4.0.jar okio-1.4.0.jar The first three Azure-specific JARs can be found in /usr/lib/hdinsight-datalake/ on the HDI head node. The Jackson JAR can be fo

This article describes various "recipes" on how to accomplish certain tasks with the NiFi processor ExecuteScript, with examples given in Groovy, Jython, Javascript (Nashorn), and JRuby. This is Part 2 in the series, I will be discussing reading from and writing to flow file contents, as well as error handling. Part 1 - Introduction to the NiFi API and FlowFiles Getting a flow file from an incoming queue Creating new flow files Working with flow file attributes Transferring a flow file Logging Part 2 - FlowFile I/O and Error Handling Reading from a flow file Writing to a flow file Reading and writing to/from a flow file Error Handling Part 3 - Advanced Features Using Dynamic Properties Adding Modules State Management Accessing Controller Services Introduction to FlowFile I/O Flow files in NiFi are made of two major components, attributes and content. Attributes are metadata about the content / flow file, and we saw how to manipulate them using ExecuteScript in Part 1 of this seri

If Kerberos for Hadoop cluster is implemented using enterprise AD, any Windows machine where users can sign-on using AD credentials has a cached ticket available. This cached ticket is available to any Windows applications by default, but any JAVA applications can't access cached ticket. To access cached ticket from a JAVA application, following registry entry in Windows should be setHKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\ParametersValueName: AllowTGTSessionKeyValueType: REG_DWORDValue:1Using klist command on Windows machine, verify username and REALM are in correct case as specified in kerberos settings on cluster. Then, create a jaas.conf file with following entryClient {com.sun.security.auth.module.Krb5LoginModule requireduseKeyTab=falseuseTicketCache=true;};To access kerborized cluster, JAVA program should be launched with following parameters.-Djava.security.auth.login.config="<path-to-jaas-conf>/jaas.conf"This will allow JAVA program to access cach

I had a hard time finding sample code online that uses the latest HBase 1.1+ Java APIs to connect to HDP 2.3 Sandbox. Here is my version for those that already know the Java basics.public static void main(String[] args) throws IOException { TableName tableName = TableName.valueOf("stock-prices"); Configuration conf = HBaseConfiguration.create(); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("hbase.zookeeper.quorum", "sandbox.hortonworks.com"); conf.set("zookeeper.znode.parent", "/hbase-unsecure"); Connection conn = ConnectionFactory.createConnection(conf); Admin admin = conn.getAdmin(); if (!admin.tableExists(tableName)) { admin.createTable(new HTableDescriptor(tableName).addFamily(new HColumnDescriptor("cf"))); } Table table = conn.getTable(tableName); Put p = new Put(Bytes.toBytes("AAPL10232015")); p.addColumn(Bytes.toBytes("cf"), Bytes.toBytes("close"), Bytes.toBytes(119)); table.put(p); Result r = table.get(new G

Short Description:Spark Hbase Connector (SHC) is currently hosted in Hortonworks repo and published as spark package. Below is simple example how to access Hbase table in Spark shell and Load the data into DataFrame. Once data is in Dataframe we can use SqlContext to run queries on the DataFrame.ArticleThe documentation here leaves out a few pieces in order access HBase tables using SHC with spark shell. Here is the Example accessing Hbase "emp" table in Spark shell. Hbase ShellCreate a simple "emp" Hbase table using Hbase shell and insert sample datacreate 'emp', 'personal data', 'professional data'put 'emp','1','personal data:name','raju'put 'emp','1','personal data:city','hyderabad'put 'emp','1','professional data:designation','manager'put 'emp','1','professional data:salary','50000'Once created exit Hbase shell and run spark shell providing SHC package and hbase-site.xml/usr/hdp/current/spark-client/bin/spark-shell --packages zhzhan:shc:0.0.11-1.6.1-s_2.10 --files /etc/hbase/conf

IntroductionThis wiki page describes the script sqoopTables.sh (GitHub: https://github.com/sourygnahtw/hadoopUtils/blob/master/scripts/sqoop/sqoopTables.sh).The purpose of the script is to sqoop in parallel many tables and to store them into Hive.MotivationsIn a previous project, I needed to download around 400 tables (out of 500) of a SQLserver database.Most of the tables were quite small (less than a few MB), which means that the overhead of fetching the metadata in Sqoop (establishing the connexions, getting the DDL of the table...) is very important compared to the time to do the real job (download the data). Getting the 400 tables could take around 6 hours.To speed up the downloading process, Sqoop usually "splits" a table in 4 parts, to parallelise the downloading process in 4 streams. In my project, such approach was not working: Many tables had not clear primary keys or the distribution of those keys was not uniform. Trying to find better split columns for 400 tables was a wast

Batch Loading with Druid and Hive IntegrationThis brief tutorial will walk through the various ways of ingesting raw data into druid and exposing the data in hive for analytics.For Reference, this tutorial was built with CentOS 7, HDP 3.0.1 w/ LLAP enabled.This tutorial will be divided into 6 sections:Data GenerationDefine the Ingest ScriptSubmit the Ingest SpecAccessing Data in HiveDouble the Data Double the FunLoading Data With HiveWhy Druid?Druid is a distributed columnar data store that is optimized for OLAP style queries. Data loaded into Druid is pre-aggregated based upon selected dimensions and metric fields. The data is further partitioned by time to enable efficient temporal access. This time based partitioning can be as long as a year and as small as a second, depending on what the business needs are.Today, Druid is tightly integrated with Hive which helps bring high speed OLAP analytics to the data lake. While Hive enhancements like LLAP deliver great performance improvement

Objective:Connect source (nifi) and destination (nifi) via site-to-site over SSL (https - 9091 port) Issue:NiFi Site to Site SSLHandShakeException PKIX path building failed - Missing TrustStore CertsRoot cause:Nifi truststores missing certificates of other Nifi Steps Taken:1. Run following command to get the list of Certificates in current Nifi TrustStore keytool -v -list -keystore <trustStoreLocation>2. If no certs that belong to target Nifi are found then we can proceed with installing the target Nifi cert3. Run following command to get the target Nifi public certificate echo -n|openssl s_client -connect <targetNiFiHostName>:9091 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/examplecert.crt3. Run following command to get the target Nifi public certificatekeytool -import -file /tmp/examplecert.crt -alias <targetNiFiCertificate> -keystore <trustStoreLocation> -storepass xxxx -noprompt 3. Verify using Openssl or SSLPoke commands java -Djavax.ne

IntroductionThis wiki page describes the script validationHiveTablesLinuxMode.sh (GitHub: https://github.com/sourygnahtw/hadoopUtils/blob/master/scripts/hive/validationHiveTablesLinuxMode.sh).The purpose of the script is to compare one Hive table with another, typically to validate a table generated by a Hive transformation against a source-of-truth (sqooped) table.MotivationsMy current project is the typical Hadoop project that consists of offloading a ETL process currently done in a relational database (SQLserver) to a Hadoop process.That involves rewriting all the (many) SQL queries in the ETL process to HiveQL queries.Some criteria of success for such project are:is the new Hive ETL workflow faster than the old one?have the output tables of the new workflow exactly the same data as in the original workflow?(does the customer pay less licenses and hardware with this new workflow?)So, getting sure that the HiveQL queries written in such ETL offloading project is a key subject. Hence

IntroductionLet's jump into tutorial 2 from my AI to Edge series!This tutorial details the creation of a Nifi flow executing the ONNX model we trained in my last article.More precisely we will try to feed these 3 handwritten digits and predict their value:Note: as always, all code/files referenced in this tutorial can be found on my github, here.AgendaBelow is an overview of the flow:As you can see, the flow is divided in the following sections:Section 1: Listening to a folder for new png filesSection 2: Resizing these images to 28x28 (size used to train our model)Section 3: Converting these images to CSV (format used to train our model)Section 4: Running our predictive modelSection 1: Listening to a folder for new png filesStep 1: Setup a variable for the root folderThis will be useful when we deploy the flow to a minifi flow. Go to your variables and create the following:Name: root_folderValue: location of your download of my githubStep 2: List files in folderCreate a ListFiles proce

Step 1 : Log into AWS your credentialsStep 2 : From the AWS console go to the following options and create a user in for the demo in AWSSecurity & Identity --> Identity and Access Management --> Users --> Create New UsersStep 3 : Make note of the credentialsawsAccessKeyId = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx';awsSecretAccessKey = 'yyyyyyyyyyyyyyyyyyyyyyyyyyy';Step 4 : Add the User to the Admin Group by clicking the button “User Actions” and select the option Add Users to Group and add select your user (admin)Step 5 : Assign the Administration Access Policy to the User (admin)Step 6 : In the AWS Console , Go to S3 and create a bucket “s3hdptest” and pick your region Step 7 : Upload the file manually by using the upload button. In our example we are uploading the file S3HDPTEST.csvStep 8 : In the Hadoop Environment create the user with the same name as it is created in the S3 EnvironmentStep 9 : In Ambari do all the below properties in both hdfs-site.xml and hive-site.xml<pro

The Log Search service may use SSL for three things:Encrypt communication between Solr and the Log Search Server / Log FeedersEncrypt communication with the portal by using https protocolEncrypt communication with the Ambari Server for authenticationIn all of these cases the Log Search Server should have a trust store / key store available which should be configured in the logsearch-env.sh file. In case #1 the Log Feeders should also have a trust store / key store available which should be configured in logfeeder-env.sh.How to configure to use SSL in these cases?For all cases you should configure the stores for the Portal (and the Log Feeders in case of #1), see below how to do it.Using SSL for the communication with Solr you should configure your Solr server to use SSL.Using https as the portal protocol you should set the property Advanced logsearch-env / logsearch_ui_protocol to https. If you are not using Ambari you should add (or update the value of) logsearch.protocol=https to the

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. This article includes a Matrix that helps developers to take advantage of the latest NiFi processor to improve old and develop new flows. The following is a table of the matrix, arranged alphabetically from K-Z:See here for the Matrix Table from A-DSee here for the Matrix Table E-JNiFi 1.5NiFi 1.4NiFi 1.3NiFi 1.2ListAzureBlobStorageListAzureBlobStorageListAzureBlobStorageListAzureBlobStorageListDatabaseTablesListDatabaseTablesListDatabaseTablesListDatabaseTablesListenBeatsListenBeatsListenBeatsListenBeatsListenGRPCListenGRPCListenHTTPListenHTTPListenHTTPListenHTTPListenLumberjackListenLumberjackListenLumberjackListenLumberjackListenRELPListenRELPListenRELPListenRELPListenSMTPListenSMTPListenSMTPListenSMTPListenSyslogListenSyslogListenSyslogListenSyslogListenTCPListenTCPListenTCPListenTCPListenTCPRecordListenTCPRecordListenUDPListenUDPListenUDPListenUDPListenUDPRecordListenUDPRecor

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. This article includes a Matrix that helps developers to take advantage of the latest NiFi processor to improve old and develop new flows. The following is a table of the matrix, arranged alphabetically from E-J:See here for the Table Matrix from A-DSee here for the Table Matrix K-ZNiFi 1.5NiFi 1.4NiFi 1.3NiFi 1.2EncryptContentEncryptContentEncryptContentEncryptContentEnforceOrderEnforceOrderEnforceOrderEnforceOrderEvaluateJsonPathEvaluateJsonPathEvaluateJsonPathEvaluateJsonPathEvaluateXPathEvaluateXPathEvaluateXPathEvaluateXPathEvaluateXQueryEvaluateXQueryEvaluateXQueryEvaluateXQueryExecuteFlumeSinkExecuteFlumeSinkExecuteFlumeSinkExecuteFlumeSinkExecuteFlumeSourceExecuteFlumeSourceExecuteFlumeSourceExecuteFlumeSourceExecuteGroovyScriptExecuteProcessExecuteProcessExecuteProcessExecuteProcessExecuteScriptExecuteScriptExecuteScriptExecuteScriptExecuteSparkInteractiveExecuteSQLExecute

Hi I want my campany name in the cloudera profile to change to xyz could any one help me with the process. 

Building an Apache NiFi processor is super easy. I have seen/read several articles on how to get started by executing maven commands via CLI. This article is geared towards individuals who like to use an IDE (specially IntelliJ) to do the imports instead of running via CLI.On IntelliJ click on create project, Check "Create from archetype", click on "ADD Archetype" and enter the followingGroupId: org.apache.nifiArtifactId: nifi-processor-bundle-archetypeVersion: <YourVersionOfNifi>and then click "OK"Now your new NiFi Archetype has been created. Select itEnter GroupId, ArtifactId, and Version of your choiceA final attribute we need to add is artifactBaseName. This is mandatory. Click on "+" and enterName: artifactBaseNameValue: whatEverYouLikeNow a project ready to build a custom processor.Enjoy!

Following are the steps to connect to Phoenix tables using Spark2.1) Create a symlink of hbase-site.xml in spark2 confln -s /etc/hbase/conf/hbase-site.xml /etc/spark2/conf/hbase-site.xml2) Launch spark-shell using phoenix spark jars in extra classpath.spark-shell --conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar" --conf "spark.driver.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar"3) Create a phoenix connection and query the tables.scala> import org.apache.spark.sql.SQLContextimport org.apache.spark.sql.SQLContextscala> val sqlContext = new SQLContext(sc)sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@495e8a3scala> val df = sqlContext.load("org.apache.phoenix.spark",Map("table" -> "TABLE1", "zkUrl" -> "localhost:2181"))df: org.apache.spark.sql.DataFr

UPDATE: This article has been vetted against HDF 2.0 - HDF 3.2. Minor updates have been made for additional clarity on use of NiFi CA for establishing trust with Ranger. Prerequisites NiFi has been installed running with SSL enabled (with certificates manually installed or using the NiFi Certificate Authority). You will need the keystore/truststore names, locations, aliases, identity (DN) and passwords used when enabling SSL for NiFi. Ensure that all nodes have the same keystore/truststore passwords applied and like named locations in order to apply ranger nifi plugin configurations consistently via Ambari. Ranger has been installed and configured with security enabled. For instructions on setting up SSL for Ranger please review Configure Ambari Ranger SSL . Note the name, location, aliases, identity (DN) and passwords used when creating keystores and truststores for Ranger If Kerberos will be used recommend that this is enabled for the HDF cluster before proceeding. Part 1 - Estab

To enable HTTPS for web HDFS, do the following: Step 1: Get the keystore to use in HDFS configurations.a) In case cert is getting signed by CA, do the following:1. Generate a keystore for each host. Make sure the common name portion of the certificate matches the hostname where the certificate will be deployed.keytool -genkey -keyalg RSA -alias c6401 -keystore /tmp/keystore.jks -storepass bigdata -validity 360 -keysize 20482. Generate CSR from above keystorekeytool -certreq -alias c6401 -keyalg RSA -file /tmp/c6401.csr -keystore /tmp/keystore.jks -storepass bigdata3. Now get the singed cert from CA - file name is /tmp/c6401.crt4. Import the root cert to JKS first. (Ignore if it already present)keytool -import -alias root -file /tmp/ca.crt -keystore /tmp/keystore.jksNote: here ca.crt is root cert5. Repeat step4 for intermediate cert if there is any.6. Import signed cert into JKSkeytool -import -alias c6401 -file /tmp/c6401.crt -keystore /tmp/keystore.jks -storepass bigdata7. I

IntroductionBefore the official release of Cloudera Data Flow (ex. Hortonworks Data Flow), you may want to play with Nifi and Hive on CDH.However, because CDH 5 is using a fork of Hive 1.1, the HiveQL processors and controller services included on the official Apache release will not work, so you need to have your own, as explained in this article: Connecting NiFi to CDH Hive.This article is awesome, but does not focus on Kerberos/SSL; since I had to do the configuration myself, I thought I would share the knowledge.Note: You could use a DBCP connection to connect to Cloudera Hive but it will not allow you to use the proper authentication.Pre-RequisitesTo connect to Hive with SSL and Kerberos, you will need the following:A running version of Nifi (I used Apache 1.9 in this example)A Kerberized CDH cluster with Hive on SSL (I used CDH 5.15 in this example)The certificate to add to your keystore for SSL connectionA keytab for a specific user authorized in the clusterThe krb5 configuratio

In the previous article, we saw how to stream tweets using NiFi, Kafka, Tranquility, Druid and Superset ...https://community.hortonworks.com/articles/177561/streaming-tweets-with-nifi-kafka-tranquility-druid.html You have to implement in that previous article, the part of Druid datasource and Nifi flow to carry on here. But life already is hard enough, why not simplify it? The idea here is to perform the same streaming but now integrating Nifi directly to Druid. So, our new diagram would look like this: As we saw in that article, we have Tranquility as an integrating factor between Kafka and Druid. Some people asked me: Why not use Kafka Indexing Service instead of Tranquility? My answer: because Tranquility as a framework, can be used flexibly, doing integration of almost any component to the Druid. Thus, on this nifi-druid integration, we will build a custom Nifi processor, which uses Tranquility to integrate data directly into Druid. Ok, it’s time to hands on! Let