Community Articles

Short Description:This article covers how to improve the performance of the NiFi UI. Article:Over time it has been seen that the users of NiFi have been building very large dataflows consisting of many thousands of components (processor, reporting tasks, controller services, etc). While NiFi in no way limits to any degree the number of components that can be added to the NiFi canvas, the more components a user adds, the less responsive the UI becomes. This processor explosion not only affects the responsiveness of the UI, but can also lead to unexpected node disconnections.--- What are the various states a component can have?NiFi components have multiple states that consist of stopped, started, enabled, and disabled. Beyond these states exists one of two statuses:Valid: Component configuration was successfully validated. This means that all required properties have been configured and in the case of processors all required connections have been accounted for (connected to another com

Before we move on to the more popular DR/HA strategies for Amabari,Hive,HBase and using Falcon/snapshot/distcp etc let us take a quick tour of the tiered storage in Hadoop. Link to Series1:https://community.hortonworks.com/content/kbentry/43525/disaster-recovery-and-backup-best-practices-in-a-t.htmlThe concept of tiered storage revolves around the idea of decoupling storage of data and the computing/performing operations on the data.Let us think about it in a more general way:You have a data lake and you are storing a 5 years’ worth of data amounting to a petabyte.You are observing the following scenarios in your cluster: Your storage has become heterogeneous with time meaning you have added new generation racks/machines over time and some of them are better performing (less dense, high spinning/ newer processor) than the others. You are also observing a trend that the last 6 months of data is getting accessed more frequently (Like 90% of the times). While the remaining 4.5 years of

Hbase works smoothly in auto pilot mode if one knows how to tune several of the knobs on its dashboard. Not only its important to understand each knob but also what its dependencies are with other knobs. There are several parameters which require tuning based on your use case or work load to make Hbase work in an optimized way. I will try to explain some of the basic parameters in this article. More advanced parameters would be covered in next article. 1. Hbase_master_heapsize To many’s surprise , a master in Hbase does not do any heavy lifting and hence never require more than 4 - 8 GB in regular setups. Master is basically responsible for meta operations such as create/ delete of tables , keeping check on region servers’ well being using watchers on zookeeper znodes , re-distribution of regions during startup (balancer) or when a region server shuts down. Please note that master's assignment manager keeps track of region states in this memory only and hence if you have huge number of

This is the continuation of article Part-1 provisioning HDP/HDF cluster on google cloud. Now that we have Google credentials created, we can provision the HDP/HDF cluster. Lets first start with HDP cluster.Login to Cloudbreak UI and click on create cluster which will open the create cluster wizard with both basic and advanced options. On the general configuration page Select the previously created Google credentials, Enter name of the cluster , Select region as shown below, Select either HDP or HDF version. For cluster type select the appropriate cluster blueprint based on your requirements. The available blueprint option in cloudbreak 2.5 tech preview are shown below.Next is configuring the Hardware and Storage piece. Select the Google VM instance type from the dropdown. Enter number of instances for each group. You must select one node for ambari server for one of the host group for which the Group Size should be set to "1".Next is setup the Network group. You can select the existing

This articles explains about how to re-run only failed action for oozie workflow. Below are the steps:1. Find out the WF id of the failed/killed job.2. Prepare a job config file which needs to be passed to the rerun command. To do so Follow below steps:2.1 You first need the oozie job configuration xml file. The easiest way to do that is to use the -configcontent option of the oozie job command. E.g. On commandlineexport OOZIE_URL="http://<oozie-host>:11000/oozie"oozie job -configcontent <workflow-id> > job_conf.xml2.2 Delete oozie.coord.application.path property from job_conf.xml. This is to avoid E0301: Invalid resource oozie rerun error.2.3 Now add below property in job_conf.xml. This determines what actions need to be run in the workflow. If we specify specific action nodes here then it will skip those actions. if nothing specified then it will run all actions of the workflow. To run all actions of a workflow:<property> <name>oozie.wf.rerun.skip.nodes<

Objective To import a versioned flow or revert local changes in a versioned flow, a user must have access to all the components in the versioned flow. As such, it is recommended that restricted components are created at the root process group level if they are to be utilized in versioned flows. This tutorial illustrates the benefits of this configuration and demonstrates a new feature introduced in Apache NiFi 1.6.0: granular restricted component categories (NIFI-4885). Users can be given access to all restricted components or to specific categories of restricted components. Note: This tutorial assumes you are familiar with setting up a secure Apache NiFi instance and integrating it with a secure Apache NiFi Registry.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.6.0 Apache NiFi Registry 0.1.0User Setup Assume the following:There are two users, "sys_admin" and "test_user" who have access to both view and modi

Cloudbreak OverviewOverview Cloudbreak enables enterprises to provision Hortonworks platforms in Public (AWS + GCP + Azure) and Private (OpenStack) cloud environments. It simplifies the provisioning, management, and monitoring of on-demand HDP and HDF clusters in virtual and cloud environments.Following are primary use cases for Cloudbreak: Dynamically configure and manage clusters on public or private clouds.Seamlessly manage elasticity requirements as cluster workloads changeSupports configuration defining network boundaries and configuring security groups.This article focuses on deploying HDP and HDF cluster on Google Cloud.Cloudbreak Benefits You can spin up connected data platform (HDP and HDF clusters) on choice of your cloud vendor using open source Cloudbreak 2.0 which address the following scenarios. Defining the comprehensive Data Strategy irrespective of deployment architecture (cloud or on premise).Addressing the Hybrid (on-premise & cloud) requirements.Supporting the k

All you need is some basic programming skill and a little experience with AWS to get things kick started.Before we proceed, let me explain what actually is Hybrid Cloud Environment?Hybrid cloud solutions combine the dynamic scalability of public cloud solutions with the flexibility and control of your private cloud.Some of the benefits of Hybrid Computing include - Business ContinuityMore Opportunity For InnovationScalabilityIncreased Speed To MarketRisk Management (test the waters)Improved ConnectivitySecure SystemsKafka MirroringWith Kafka mirroring feature you can maintain a replica of you existing Kafka Cluster.The following diagram shows how to use the MirrorMaker tool to mirror a source Kafka cluster into a target (mirror) Kafka cluster.The tool uses a Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the local (target) cluster using an embedded Kafka producer Use CaseDemonstrate hybrid cloud solution using Kafka Mirroring across regi

Using a Pre-Trained Neural Network on MS CoCo DataSet using Mask R-CNN in TensorFlow / Keras. I am always looking for use cases, always. Yesterday I gave a talk at the IoT Fusion Conference in Philadelphia about IoT. I had a Raspberry Pi taking web camera images. I mentioned I wasn't pointing at people because they may not want to be photographed. I am concerned with privacy. It gave me the idea it would be cool to block people or other things from images. Then on queue, a great library shows up in github. Thanks to Minimaxir's Person Blocker, I can now use Apache NiFi to pull remove people from images. I did a couple of minor tweaks to his code to add OpenCV image capture and output some JSON information on what happened. I am running this on an OSX laptop, but at some point I'll move it to a Raspberry Pi, TinkerBoard or NVidia Jetson TX1. Please support this project: https://github.com/minimaxir/person-blocker https://www.patreon.com/minimaxir Using this

SymptomsWhen attempting to access a running cluster via SSH, the connection times out or is rejected Applies ToAll versions of clusters managed by Cloudera Altus CauseUpdate needed for AWS Security group, as the incoming SSH connection originating for client machine may not be included on the defined cluster's AWS Security Group incoming connections     Troubleshooting StepsThis issue can typically occur if the AWS security group configured for does not contain an updated IP from the connecting client.   To alleviate this issue, please perform the following steps which will add the IP address(es) needed to connect to the appropriate AWS security group: Login to AWSSelect the Appropriate Region in the top-right hand corner, (i.e. US West (Oregon)).Navigate to EC2Under Resources, click on Security GroupsLocate the corresponding Cloudera Altus Security Group configured;  For a particular cluster in question, narrow down the options filtering by&nbsp

Operational Challenges Legacy applications that lack the ability to leverage the REST interface for HDFS can limit data access for storage.One of the ways to deal with this is a mountable filesystem method for Hadoop. Additions to Linux kernel for native HDFS access are not yet there. Therefore we need to use the “user space” capabilities with something like FUSE to provide this functionality. https://en.wikipedia.org/wiki/Filesystem_in_UserspaceBut before we begin, we need to understand some limitations of the HDFS-FUSE implementation: Because this runs in user space (vs kernel space), performance is not on par with what native API implementations can provide For large data transfers, continuous ingest, and in particular - streaming operations, users should look at solutions like Apache NIFI And last but not least, the use of HDFS-FUSE does not currently support “append” operations for a file So with those limitations understood, lets begin getting things setup on our cluster.Installa

The script provided here can be used to perform health check of a node before installing DSX. There are certain pre-requisites when installing IBM DSX, without which installation cannot go through. This script helps to validate health of a node and check if it is fit to install DSX.Note: Make changes to line 166 and 182 as per your installation. The drives may vary based on where you are doing the installation.# /bin/bashfunction checkRAM(){ local size="$1" local limit="$2" if [[ ${size} -lt ${limit} ]]; then echo "WARNING: RAM size is ${size}GB, while requirement is ${limit}GB" | tee -a ${OUTPUT} return 1 fi}function checkCPU(){ local size="$1" local limit="$2" if [[ ${size} -lt ${limit} ]]; then echo "WARNING: CPU cores are ${size}, while requirement are ${limit}" | tee -a ${OUTPUT} return 1 fi}function usage(){ echo "This script checks if this node meets requirements to install DSX-Local. " echo "Arguments: " echo "--type=[9nodes_master|9nodes_storage|9nodes_comp

Guide to deploying a 3-node DSX Local Cluster on AWSPart 1: Deploying AWS ClusterOn your EC2 dashboard click on launch instance. Step 1: Choose an Amazon Machine Image (AMI) Select AWS Marketplace and search for Centos -> Chose the Centos version you need (I chose CentOS 7 (x86_64) - with Updates HVM) Step 2: Choose an Instance Type. Select m4.4xlarge or m4.2xlarge Step 3: Configure Instance Details Set Number of Instances as 3Step 4: Add Storage You need to add Root storage minimum 50GB and additional EBS storage of around 500GB( if you deploying for just test purpose 300GB is fine). DSX usually require high IOPS so provisioned SSD are preferred. But be advised that provisioned SSD are expensive due to high IOPS.Step 5: Add Tags Optional Step 6: Configure Security Group You need to set up a custom security group, which should include - Custom TCP rule, port 6443 and range 172.31.0.0/16 -> used for internal connectionAllow all connection between nodes Allow restricted connection

Due to a recent change in Google Cloud's handling of requests, all clusters created on Google Cloud Platform with Cloudbreak Cloudbreak 2.4.0 (or earlier 2.x) and 1.16.5 (or earlier 1.x) fail with an error similar to:Infrastructure creation failed. Reason: com.sequenceiq.cloudbreak.cloud.gcp.GcpResourceException: Error during status check: [ resourceType: GCP_DISK, resourceName: hdf-test, stackId: 5, operation: hdftest-s-2-20180323123311 ] [ Cause message: 403 Forbidden { "code" : 403, "errors" : [ { "domain" : "global", "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'", "reason" : "forbidden" } ], "message" : "Required 'compute.globalOperations.get' permission for 'projects/*******/global/operations/operation-1521808391647-56813a099af18-300f33eb-8a28c20c'" } ]In order to address the issue, we created a 2.4.1 maintenance release. You can upgrade to Cloudbreak 2.4.1 by using

Errororg.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.Underlying cause: java.sql.SQLException : Access denied for user 'hive'@'node1.hortonworks.com' (using password: YES)SQL Error code: 1045org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version. at org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:80) at org.apache.hive.beeline.HiveSchemaTool.getConnectionToMetastore(HiveSchemaTool.java:133) at org.apache.hive.beeline.HiveSchemaTool.testConnectionToMetastore(HiveSchemaTool.java:187) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:277) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:526) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(

In this case increasing the Ulimit of number of processes to the Druid user will fix this issue.This Article explains more the issue and best way to fix such issue.Exception stack.2018-04-02T23:41:56,827 ERROR [main] io.druid.cli.CliBroker-Errorwhen starting up.Failing.java.lang.OutOfMemoryError: unable to create newnative thread at java.lang.Thread.start0(NativeMethod)~[?:1.8.0_40]at java.lang.Thread.start(Thread.java:714)~[?:1.8.0_40]at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)~[?:1.8.0_40]at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)~[?:1.8.0_40]at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)~[netty-3.10.6.Final.jar:?]at or

Reading and writing files to a MapR cluster (version 6) is simple, using the standard PutFile or GetFile, utilizing the MapR NFS. If you've searched high and low on how to do this, you've likely read articles and GitHub projects specifying steps. I've tried these steps without success, meaning whats out there is too complicated or out-dated to solve NiFi reading/writing to MapR. You don't need to re-compile the HDFS processors with the MapR dependencies, just follow the steps below:1) Install the MapR client on each NiFi node#Install syslinux (for rpm install)sudo yum install syslinux#Download the RPM for your OS http://package.mapr.com/releases/v6.0.0/redhat/ rpm -Uvh mapr-client-6.0.0.20171109191718.GA-1.x86_64.rpm#Configure the mapr client connecting with the cldb/opt/mapr/server/configure.sh -c -N ryancicak.com -C cicakmapr0.field.hortonworks.com:7222 -genkeys -secure#Once you have the same users/groups on your OS (as MapR), you will be able to use maprlogin password (allowing y

I want to easily integrate Apache Spark jobs with my Apache NiFi flows. Fortunately with the release of HDF 3.1, I can do that via Apache NiFi's ExecuteSparkInteractive processor.First step, let me set up a Centos 7 cluster with HDF 3.1, follow the well-written guide here.With the magic of time lapse photography, instantly we have a new cluster of goodness:It is important to note the new NiFi Registry for doing version control and more. We also get the new Kafka 1.0, updated SAM and the ever important updated Schema Registry.The star of the show today tis Apache NiFi 1.5 here. My first step is to Add a Controller Service (LivySessionController).Then we add the Apache Livy Server, you can find this in your Ambari UI. It is by default port 8999. For my session, I am doing Python, so I picked pyspark. You can also pick pyspark3 for Python 3 code, spark for Scala, and sparkr for R.To execute my Python job, you can pass the code in from a previous processor to the ExecuteSparkInt

PROBLEM: In hive, on oracle metastore following error is observed for table creation,java.sql.SQLException: ORA-01461: can bind a LONG value only for insert into a LONG column ROOT CAUSE: Issue looks to be due to large number of columns in this table. RESOLUTION: Currently hive will we store column stats in a table while we store the accuracy in another table (table properites). The best way is to store both the column stats and its accuracy in the same table. This involves modification of schema.As a workaround we can change the column type to CLOB.

DevOps: Backups Part 2Apache NiFi Registry Rest API Docshttps://nifi.apache.org/docs/nifi-registry-docs/rest-api/index.htmlGithub:https://github.com/tspannhw/BackupRegistry

http://iotfusion.net/session/enterprise-iiot-edge-processing-with-apache-nifi-minifi-and-deep-learning/Join me for this live in Philly on April 5, 2018.Sending Data to an Advanced Analytics Platform like https://www.zoomdata.com/ is childsplay. I added that as a live stream via REST while I am sending the JSON to be converted to AVRO then ORC for storage and Hive queries.As part of the ingest we store the images and make a current image for display.See: https://community.hortonworks.com/articles/182850/vision-thing.html https://community.hortonworks.com/content/kbentry/182984/vision-thing-part-2-processing-capturing-and-displ.html An Overview of a Flow to Ingest Both Apache MXNet + SenseHat Data plus multiple rows of TensorFlow dataIf TensorFlow json data, split it into individual JSON recordsThis is how to split an array of JSON that doesn't have a top element.Reference:https://community.hortonworks.com/articles/182850/vision-thing.htmlGithub:https://github.com/tspannhw/OpenSource

Properly Size IndexUnderstanding what to index often requires deep business domain expertise on the data. This yields better indexing strategy and increases accuracy for searching data. Not all data will be indexed but for an organization who's acquiring brand new data, this requires indexing all data until it is understood what value it brings to the business. What this means is that data needs to be re-indexed so it is a good practice to store raw data somewhere cheap, often in HDFS or in the cloud object storage. Tuning for SpeedThis starts with monitoring QTime. It provides performance metrics on how fast the request was received, query parsing and actual search. This doesn't include the time it took to send back the response to the client which depends on heavily on how big the payload and how fast the network I/O is. Sorting works best with short valued properties like price, age, etc but doesn't on tokenized values like date, long field type values and others. For range queries,

An article on the challenges and solutions to predicting machine failures in the field. The full details can be found here: https://github.com/kirkhas/zeppelin-notebooks/tree/master/Preventive_maintenanceStep #1 Feature SelectionStep #2 GeolocationStep #3 - Scythe is a time-series library authored by Kirk Haslbeck for these purposes- Needed to Resample the data into trips or route segments (Scythe Resample)- Needed to Step Interpolate the miles since last service to be 4K, 5K and less continuous regression Step #4 - Indexing and OneHotEncoding to the Rescue. Found a relationship of a particular "Make" that was more problematic than most.Roc Curve - A near perfect model

PARTITIONS IN HIVE Hive tables are organized and stored as partitions. It helps improve the query performance. By default, Hive stores each partitions data in a separate folder under the table’s locations in the file system. Hive also provides support for individually setting the location of each partition. This allows users to store hive partitioned data in different locations. Let us take an example of a hive table called student with the four columns - id, name, dept and year and partitioned on the year column. Let’s say we insert data into this table for the years 2009 and 2011. By default, the hive table data is stored under the hive metastore directory in HDFS specified by the parameter $hive.metastore.warehouse.dir in hive-default.xml. Fig.1 shows the two partition folders under the student directory in the hive metastore. Fig.1. Partitions in default location in Warehouse. Now, let us say we want to add a partition for the year 2018 but store the data in a different l

As discussed in my previous article Using Pre-defined Security Groups with Cloudbreak, the preferred method for managing security for hostgroups in Cloudbreak is to use the native interface from your cloud provider. For Openstack, one of the easiest ways to do it is to use the openstack CLI. My Openstack environment uses a self-signed certificate and is only accessible through a VPN, so I have to use the --insecure flag to ignore the certificate errors. Before you can use the CLI commands, you have to know your Openstack login parameters. This includes username and password, but also the URL of the security endpoint, project name, etc. For mine, I have a script that puts these values in environment variables, but they can be specified on the openstack command line if desired. export OS_USERNAME=john.whitmoreexport OS_PASSWORD='xxxxxxxxxxxxxx'export OS_SYSTEM_SCOPE=Projectexport OS_USER_DOMAIN_NAME=Defaultexport OS_AUTH_URL=http://###.###.###.###:5000/v3export OS_IDENTITY_API_VER