Resources

Posting Images with Apache NiFi 1.7 and a Custom ProcessorI have been using a shell script for this since Apache NiFi did not have a good way to natively post an image to HTTP servers su such as the model server for Apache MXNet.So I wrote a quick and dirty processor that posts an image there and gathers the headers, result body, status text and status code and returns them to you as attributes.In this example I am download images from picsum.photos free photo service.To use this new processor, download to your lib directory and restart Apache NiFi, then you can add the PostImageProcessor.Eclipse For Building My ProcessorConfigure the Post Image Processor with your URL, fieldname, imagename and image type.MXNet Model Server ResultsThe Attribute Results From the DataResultsExample Resultspost.header{Server=[Werkzeug/0.14.1 Python/3.6.6], Access-Control-Allow-Origin=[*], Content-Length=[396], Date=[Fri, 05 Oct 2018 17:47:22 GMT], Content-Type=[application/json]}post.results{"prediction":

Short Description:Configure Knox to access Atlas UIArticleHere is the steps to access Atlas UI through Knox. 1. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.6.****/services3. mkdir –p atlas/0.8.0/ mkdir –p atlas-api/0.8.0/4. download the configurations from https://github.com/apache/knox/tree/v0.13.0/gateway-service-definitions/src/main/resources/services/atlas-api/0.8.0 URL to /var/lib/knox/data-2.6.***/services/ atlas-api/0.8.0/5. download the configurations from https://github.com/apache/knox/tree/v0.13.0/gateway-service-definitions/src/main/resources/services/atlas/0.8.0 URL to /var/lib/knox/data-2.6.***/services/ atlas/0.8.0/5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.6.**/services/atlas*/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service><role>ATLAS</role><url>sandbox.hortonworks.com:21000</url>&

Most data movement use cases do not require a “shuffle phase” for redistributing FlowFiles across a NiFi cluster, but there are few cases where it is useful. For example:ListFile -> FetchFileListHDFS -> FetchHDFSListFTP -> FetchFTPGenerateTableFetch -> ExecuteSQLGetSQS -> FetchS3In each case, the flow starts with a processor that generates tasks to run (e.g. filenames) followed by the actual execution of those tasks. To scale, tasks need to run on each node in the NiFi cluster, but for consistency, the task generation should only run on the primary node. The solution is to introduce a shuffle (aka load balancing) step in between task generation and task execution. Processors can be configured to run on the primary node by going to “View Configuration”-> “Scheduling” and selecting “Primary node only” under “Execution”. The shuffle step is not an explicit component on the NiFi canvas, but rather the combination of a Remote Input Port and a Remot

Objective The objective of this article is to present a workflow to capture and republish data in Kafka topics. Note that NiFi is NOT a Kafka Replication/Backup tool. This is just a workflow based approach to capture kafka data and publish it back to a fresh topic if necessary. Although storage of Kafka data is not mandatory is several cases, it may be essential in cases such as Archival – Dump Kafka Data in HDFS Corrections/Inspections – In cases where an invalid event may disrupt a downstream system, to correct and restore the messages. Start by creating two processor groups – One for the backup workflow, second for the restore workflowBackup Workflow Backup is a long running workflow which archives data to HDFS. A file in HDFS may contain a batch of events or a single event, based on how frequently messages are getting published to the source kafka topic. Hence, user does not have to stop/start this backup workflow frequently.Restore Workflow Restore a manually triggere

Properties File Lookup Augmentation of Data Flow in Apache NiFi 1.7.xA really cool technologist contacted me on LinkedIn and asked an interesting questionTim, How do I read values from a properties file and use them in my flow. I want to update/inject an attribute with this value. If you don't want to use the Variable Registry, but want to inject a value from a properties file how to do it. You could run some REST server and read it or does some file reading hack. But we have a great service to do this very easily!In my UpdateAttribute (or in your regular attributes already), I have an attribute named, keytofind. This contains a lookup key such as an integer or a string key. We will find that value in the properties value and give you that in an attribute of your choosing.We have a Controller Service to handle this for you. It reads from your specified properties file. Make sure Apache NiFi has permissions to that path and can read the file.PropertiesFileLookupService h

Overview Time to continue my series on news author personality detection! This tutorial will look at data science, Zeppelin notebooks with the idea of predicting the popularity of an article based on the personality of an author. This tutorial does assume that you completed part 1 and part 2 of the series. After completing this part, your architecture will look like this: This tutorial will be divided in sections: Section 1: Update Nifi to send article data to Phoenix Section 2: Read Phoenix data with Spark in Zeppelin Note Section 3: Train Linear Regression Model Section 4: Make and store article popularity predictions Note: This exercise is a very simplified way to apply machine learning. I spent very little time selecting the appropriate algorithm, accumulating enough data or even challenging whether or not the personality of an author is a viable predictor for personality. This is a full time job, it's called being a data scientist! But hopefully this will enable

We have seen performance issue and stability issue in heavy workload system, this is due to disk latency/Shared Disk. I.e frequent namenode failover, longer boot time, slower checkpoint, slower logging, Higher fsync will cause session expiry, etc. We never recommend a shared disk for Namenode, Journal node and Zookeeper. All of these services should have a dedicated disk. You can configure following disk type according to your HDFS workload.[dfs.namenode.name.dir] Namenode fsimage directory, dedicated disk => HDD 15K RPM [dfs.journalnode.edits.dir] Namenode edit log directory, dedicated disk => SSD [dataDir from zoo.cfg] Zoookeeper snapshot and transaction logs, for normal usage from ZKFC and HBase => HDD 15K RPM Zoookeeper snapshot and transaction logs, If it used by Nifi/ Accumulo/ Kafka/ Storm/ HBase and ZKFC => SSD If you are using RAID for meta-directory(dfs.namenode.name.dir & dfs.journalnode.edits.dir), then disable RAID and check the Non-RAID performance. Ther

Introduction With HDFS HA, the NameNode is no longer a single point of failure in a Hadoop cluster. However the performance of a single NameNode can often limit the performance of jobs across the cluster. The Apache Hadoop community has made a number of NameNode scalability improvements. This series of articles (also see part 2, part 3, part 4 and part 5) explains how you can make configuration changes to help your NameNode scale better and also improve its diagnosability. Audience This article is for Hadoop administrators who are familiar with HDFS and its components. If you are using Ambari or Cloudera Manager you should know how to manage services and configurations. HDFS Audit Logs The HDFS audit log is a separate log file that contains one entry for each user request. The audit log is a key resource for offline analysis of application traffic to find out users/jobs that generate excessive load on the NameNode. Careful analysis of the HDFS audit logs can also identify app

The service RPC port gives the DataNodes a dedicated port to report their status via block reports and heartbeats. The port is also used by Zookeeper Failover Controllers for periodic health checks by the automatic failover logic. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages.Steps:1) Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site -> Add Propertydfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenode1 host:rpc port>dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenode2 host:rpc port>dfs.namenode.service.handler.count=(dfs.namenode.handler.count / 2)This RPC port receives all DN and ZKFC requests like block report, heartbeat, liveness report, etc.. Example from hdfs-site.xml,dfs.nameservices=shva dfs.internal.nameservices=shva dfs.ha.namenodes.shva=nn1,nn2 dfs.namenode.rpc-address.shva.nn1=hwxunsecure2641.openstackloc

Objective -To store multiple row versions in HBase to evaluate the impact on performance when doing reading all versions vs. getting the latest version. To put this differently, would storing multiple versions affect the performance when querying the latest version.-Using NiFi to be able to quickly ingest millions or rows into HBaseWarning -Do not store more than a few versions in HBase. This can have negative impacts. HBase is NOT designed to store more than a few versions of a cell.Step 1: Create Sample Workflow using NiFi to ingest data into HBase table -Create HBase TableDataset: https://www.citibikenyc.com/system-datacreate 'venkataw:nycstations','nycstationfam'0 row(s) in 1.3070 secondshbase(main):014:0> desc 'venkataw:nycstations'Table venkataw.nycstations is ENABLEDvenkataw.nycstationsCOLUMN FAMILIES DESCRIPTION{NAME => 'nycstationfam', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE',

1. Create keystore file first.keytool -genkey -keyalg RSA -alias ooziehost -keystore keystore.jks -validity 360 -keysize 20482. Login to Ambari server and go to Oozie configurations. 3. Add below configurations under "Advanced oozie-env"export OOZIE_HTTPS_PORT=11443export OOZIE_HTTPS_KEYSTORE_FILE=/tmp/keystore.jksexport OOZIE_HTTPS_KEYSTORE_PASS=bigdata4. Restart Oozie service from UI.5. Now launch the Oozie UI from quick links - you should be able to access the URLhttps://OOZIEHOST:11443/oozie/Note: Make sure to import the cert into Ambari Truststore.

If you have seen my article on Microservice on Hive, this is the Phoenix version. Phoenix seems to be a better option for REST microservice. I like having HBase as my main data store, it's very performant and highly scalable for application style queries.See: https://community.hortonworks.com/articles/53629/writing-a-spring-boot-microservices-to-access-hive.htmlThis microservices is a Spring Boot REST service on top of the data loaded by this NiFi data flow.See: https://community.hortonworks.com/content/kbentry/54947/reading-opendata-json-and-storing-into-phoenix-tab.htmlPom<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.dataflowdeveloper</groupId> <artifactId>phoenix</artifactId> <version>

The following link has some sample code to connect to a secure Hive Server2 via the default connection string and via knox. https://github.com/vspw/hiveJDBCIt uses the following Jar files:-commons-configuration-xx.jar (UGI set configurations and metrics)commons-logging-xx.jar (*Not mandatory)hadoop-auth-xx.jarhadoop-common-xx.jar (UGI stuff, hadoop configurations)hive-jdbc-1.2xx-standalone.jarlog4j-xx-api-xx.jar (for log4j)log4j-api-xx.jar (for log4j)log4j-core.xx.jar (for log4j)xercesImpl-xx.jar It also reads a bunch of properties like "keytabLocationWindows", "jdbcConnStringDirect", "hiveQuery" etc. from a properties file and logging is enabled with log4j2.xml.The class: "HiveJDBCKnox" , has some instructions to make knox connectivity work on windows platform as well.

Got error [User home directory not found] when creating WorkFlow Manager view in AmbariResolve:sudo -u hdfs hdfs dfs -mkdir /user/adminsudo -u hdfs hdfs dfs -chown admin /user/adminResule:

SYMPTOMOozie web UI doesn't work on Internet explorer in Kerberized environment, it gives below error message even after making sure that Ext JS libraries are installed correctly.Oozie web console is disabled.To enable Oozie web console install the Ext JS library. ROOT CAUSEhttps://issues.apache.org/jira/browse/OOZIE-2322 - Because of empty space as a value of oozie.authentication.cookie.domainWORKAROUND1. Set value of oozie.authentication.cookie.domain to name of your corporate domain in oozie-site.xml via Ambari UI.Example:<property><name>oozie.authentication.cookie.domain</name><value>hortonworks.com</value></property>2. Stop Oozie server via Ambari.3. Move oozie.war to some other location.mv /usr/hdp/current/oozie-server/oozie-server/webapps/oozie.war /root/4. Move Oozie webapp directory to some other location. mv /usr/hdp/current/oozie-server/oozie-server/webapps/oozie /root/5. Regenrate Oozie war file using below command/usr/hdp/current/oozie-s

Related: ApacheCon 2018 in MontrealWith my talk in a dual language city, I thought perhaps I should do my talk in French. My college French is very rusty and my accent is very New Jersey. The two don't mix well. So let's have Apache NiFi do it for me. After publically debating this on Twitter, I decided to see if I could implement a solution.Secret:Most of the heavy lifting is done by Python which calls Google Translate API under the covers, automagically.My Presentation is here: https://www.slideshare.net/bunkertor/apache-deep-learning-101-apachecon-montreal-2018-v031Flow to Extract FrenchApache Tika extracts the text from PDF or PPTX and converts to text or HTML. ( I chose text). Run the Translate Python with a sentence extracted from the PDF or PPTX.Let's send that translated french line to it's own slack channel.Let's send the english to another.And there it is:runtranslate.shpython3.6 -W ignore /Volumes/TSPANN/2018/talks/IOT/translate.py "$1" 2>/dev/nulltranslate.py

The Cloudera Community has been helping users find answers to their Hadoop questions for over three years now. Over that time the community, along with the Hadoop ecosystem, has grown and changed in many ways. As a result, we are assessing the current and future state of this community to ensure that we are offering an environment that best meets the needs of our users. This is where we need your help. We are opening this thread to collect your feedback on what you like, what can be improved and any new features you would like on the community.   Some potential areas we would like feedback on are:   Segmentation of the forums - Currently the boards are mostly based on components and their function.   This makes it difficult to know where to post your new question and can also reduce the number of people who see your topic. Discussion style: is the current discussion thread style working for you?  Are there examples of other communities with a more intuit

Some times though moving master wizard is completed still UI shows the message as "Move Master Wizard In Progress".To fix this problem you can follow below steps.Step-1)curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"reassignMasterController\"}"}'http://ambari.example.com:8080/api/v1/persistStep-2). Login to the Ambari UI with "admin" user.Step-3). Create a new "local" user with some name like "admin1" (any password like "admin1")Step-4). Open another browser and login with the "admin1" user. User might see that the "Add Service Wizard in Progress" link is blinking at the top.Step-5). Now from the earlier browser where we logged in with "admin" user. We will need to click on the "Actions" => "Add Service"Step-6). Now if the "admin" user does nothing. Or Shuts down his/her laptop. Still the other admin users (like "admin1") will not be able to perform any action in the ambari UI and will keep seeing the "Add

During upgrade, if Namenode restarts timeout, it may not appear to be a problem as the request times out from the Ambari UI but the restart process continue to run in the background.However, this can lead to inconsistencies in Ambari database and cause further issues at Finalize upgrade step.Note: This article is only useful upto Ambari-2.5.x version and must be performed before starting the upgrade process.Ambari-2.6 onwards it is a one step change where you only need to update the ambari.properties file instead of all the xml changes listed below.For Ambari-2.6 onwards: refer this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/preparing_to_upgrade_ambari_and_hdp.htmlCaution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risks before following the steps.You can increase the timeouts for namenode restart using the f

Using GluonCV 0.3 with Apache MXNet 1.3 source code: https://github.com/tspannhw/nifi-gluoncv-yolo3 *Captured and Processed Image Available for Viewing in Stream in Apache NiFi 1.7.xuse case:I need to easily monitor the contents of my security vault. It is a fixed number of known things. What we need in the real world is a nice camera(s) (maybe four to eight depending on angles of the room), a device like an NVidia Jetson TX2, MiniFi 0.5 Java Agent, JDK 8, Apache MXNet, GluonCV, Lots of Python Libraries, a network connection and a simple workflow. Outside of my vault, I will need a server(s) or clusters to do the more advanced processing, though I could run it all on the local box. If the number of items or certain items I am watching are no longer in the screen, then we should send an immediate alert. That could be to an SMS, Email, Slack, Alert System or other means. We had most of that implemented below. If anyone wants to do the complete use case I can assist.demo i

Hi, I'm watching 'On Demand' webinars, but saw 'Speed' button missing.This is slightly inconvenience when you want to watch webinar with 1.25x or 1.5x speed. Wondering, if someone from respective department addresses this issue. Sample webinar: https://www.cloudera.com/content/dam/www/marketing/resources/webinars/smarthub-on-cloudera-webinar.png.landing.html  

This article is for those who want a cheat sheet for a smooth installation of HDP in a Dev, or Test with one or more of the following requirements: Place all the log data into a different directory, not /var/log All your service user names must be prefixed with the cluster name. The requirement is that these users must be centrally managed by AD or an LDAP. You do not have any local users in the Hadoop cluster, including Hadoop service users. This becomes important if you wish to have Centrify deployed also, or if you would be deploying multiple clusters with a single LDAP/ AD integration. Once again, these service names should have a cluster-prefix. You want to set appropriate YARN, Tez and MapReduce, Amabri Metrics Memory Parameters during Install. Side Note: It is always prudent to get Professional Service assistance to either install or configure your production deployment, to make sure all the per-requisites, unique to your environment are covered and met. ----------------------

One quirk of Apache Phoenix when compared to traditional RDBMS is that Phoenix provides no notion of simple username/password based authentication. This largely stems from Apache HBase, which Phoenix is built on, also not providing this as a form of authentication. With the introduction of the Phoenix Query Server, we have a number of new means which can be used to interact with Phoenix. We also have the ability to hook together new systems to provide features, like username/password authentication, which are not traditionally supported. There are multiple products available which can perform this kind of authentication, but we can trivially show that this works via a common HTTP load balancer, HAProxy. Let's assume that we have the Phoenix Query Server running on our local machine listening on the standard 8765 port. We can enable some trivial authentication using HAProxy. First, we need to create our HAProxy configuration file. global maxconn 256defaults mode http option redisp

The quickest way to launch a cluster with Cloudbreak is through the CLI. This allows you to specify all of the required settings and submit the request to Cloudbreak. The cluster will show up in Cloudbreak just like it does when launched via the wizard. The quickest way to generate the JSON template is to set up a cluster through the wizard.Generating the JSON Template from Cloudbreak GUIThe first way you can create the template is from the Create Cluster wizard. In the Cloudbreak GUI, walk through all of the steps to configure a cluster. At the end, before you click "Create Cluster" you can click "Show CLI Command". The second way is to go the a running cluster in the Cloudbreak GUI. Click the "Actions" button and you can click "Show CLI Command".The CLI command dialog comes up with the JSON template in the top text box and the Cloudbreak CLI command in the lower text box. Click the "Copy the JSON" button and the paste it into a text file. Copy the CLI command to a note for la

This article explains how to delete a Registered HDP cluster from DPSSteps:1. Download the attached file dp-cluster-remove.txt and store it in postgres db server. 2. psql -f ./dp_cluster_remove.txt -d dataplane -v cluster_name=c149-d --> <dataplane database name>cluster_name --> Your HDP clustername that should be removed from DPS cluster.Example Output:-[root@rraman bin]# docker exec -it dp-database /bin/bash bash-4.3# su - postgres de0ff40ad912:~$ ls -lrt total 8 drwx------ 19 postgres postgres 4096 Sep 21 01:27 data -rwxrwxrwx 1 postgres postgres 892 Sep 21 17:59 dp_cluster_remove.txtde0ff40ad912:~$ psql -f ./dp_cluster_remove.txt -d dataplane -v cluster_name=c149 CREATE FUNCTION remove_cluster_from_dps ------------------------- (1 row) DROP FUNCTION