Community Articles

So I found another low-end affordable platform from China for running MiniFi goodness. For this 512MB RAM machine, I decided to use MiniFi CPP. git clone https://github.com/apache/nifi-minifi-cpp.gitapt-get install cmake gcc g++ bison flex libcurl-dev librocksdb-dev librocksdb4.1 uuid-dev uuid libboost-all-dev libssl-dev libbz2-dev liblzma-dev doxygen -yapt-get install -y libleveldb-devapt-get install -y libxml2apt-get install libpython3-dev -yapt-get install liblua5.1-0-dev -yapt-get install libusb-1.0.0-0-dev libpng12-dev -yapt-get install docker.io python-virtualenv -y apt-get install libpython3-dev -yapt-get install libgps-dev -yapt-get install libpcap-dev -yapt-get install cmake gcc g++ bison flex -y ./bootstrap.shinteractive UIcd nifi-minifi-cpp-0.3.0-source/mkdir buildcd buildcmake ..makemake packageapt-get install libssl-dev -- The following features have been enabled: * EXPRESSION LANGUAGE EXTENSIONS , This enables NiFi expression language * H

Using SiteToSiteProvenanceReportingTask to Send Provenance to Apache NiFi for Processing. Eating our own provenance food!It's almost comically easy to do this. You set up a task on the server you are reporting on that sends the data to your receiver. That other server you make a simple flow to ingest and process that. I stored it to HBase as JSON as it's a good place to put a lot of data fast. Send The DataYou need to create a SiteToSiteProvenanceReportingTask in Controller Settings - Reporting Tasks. It's pretty simple. Set the values above with your destination NiFi server and a port name that you have created already. Receive the Data and Process An Individual JSON Record Split the JSON into Records $.[*] Save to HBase (PutHBaseJSON) First I have to create a table.hbase shellHBase Shell; enter 'help<RETURN>' for list of supported commands.Type "exit<RETURN>" to leave the HBase ShellVersion 1.1.2.2.6.2.0-205, r5210d2ed88d7e241646beab51e9ac147a9

Prior Ranger Version 0.6 it was a common scenario that security admins created per users policies, e.g. grant a user access to project databases (hive) or special HDFS directories. Initially, a policy was created for each user so the admins ended up to manage about thousands of policies. Especially HDFS policies often had the name in the path, e.g. /user/demo/USERNAME/ *.As Ranger 0.6 introduces the user variable, now the security admin have only to create one policy that looks like this in the path: /user/demo/{USER}/ * and assign the appropriate user permissions for the directory.The user can immediate access and use the new HDFS directory with enforced user permissions. dummy_1$ hdfs dfs -copyFromLocal tst.x1 /user/demo/dummy_1dummy_1$ hdfs dfs -ls /usr/demo/dummy_1Found 1 items-rw-r--r-- 3 dummy_1 hdfs19001 2018-03-23 15:36 /user/demo/dummy_1/text.txtWith the use screen-shot-2018-03-23-at-165124.pngof user variables now the security administrators can create more dynamic policies

Simple illustration of locking in Hive when ACID is enabledConsiderations for illustrationCluster Version: HDP -2.5.6.0Hive Version: Hive 1.2.1000Enabled with following properties in place hive.support.concurrency=truehive.compactor.initiator.on=truehive.compactor.worker.threads=1hive.exec.dynamic.partition.mode=nonstricthive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager (if non-Ambari cluster)Types of Supported Locks S = SHARED or SHARED_READX = EXCLUSIVETables used for testingorc_tab (ORC format table with col1 int and col2 string), non-transactional orc_tab_bucketed(ORC format table with col1 int and col2 string, transactional)txt_tab (TEXT format table with col1 int, col2 string, non-transactional, for loading purposes)Either tables have closed to 5 GB data on a Single node clusterSCENARIO 1 (Non Transactional Table)SELECT blocks ALTERSELECT starts first followed by ALTER Session Abeeline -u "jdbc:hive2://localhost:10000/default" -n hive -p '' -e "select col1 from orc_

Our scenario for this walkthrough is as follows: we have a customer table that contains fields for Zip Code, MRN, and Blood Type. Per policy, users in the analyst group cannot access MRN and Blood Type together with Zip Code within the same query, as this would deanonymize sensitive Personal Health Information.In order to make use of Ranger functionality to achieve this, we'll need to register a new policy condition using the Ranger API. Please see my HCC post on this topic for further details. In this case, the policy condition will have the following form (please note itemId value is specific to one's environment) within the policyConditions array contained in the /servicedef/name/hive resource:{ "itemId": 1, "name": "resources-accessed-together", "evaluator": "org.apache.ranger.plugin.conditionevaluator.RangerHiveResourcesAccessedTogetherCondition", "evaluatorOptions": {}, "label": "Resources Accessed Together?", "description": "Resources Accessed Toget

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

Objective This article highlights some of the latest UI enhancements added in Apache NiFi 1.5.0.Environment The examples shown in the article utilized the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.5.0"Primary Node" Processors Identification In a NiFi Cluster, processors that have been configured for "Primary node" execution are now identified in the UI by a "P". On the canvas, the "P" is visible next to the processor icon: The "P" is also shown in the Processors tab on the Summary page, specifically in the Name column:Finding Processors Quickly in the Summary Page If your flow has hundreds of processors, it can be difficult differentiating between them in the Summary page (accessible from the top-right Global menu). On the Processors tab, a "Process Group" column has been added to display the name of the parent process group containing the component: Additionally, when hovering over the "Go to location" button the tooltip now includes the path of t

In previous post we have seen how to install single node HDP cluster using Ambari Blueprints. In this post we will see how to Automate HDP installation using Ambari Blueprints. .Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Below are simple steps to install HDP multinode cluster using internal repository via Ambari Blueprints..Step 1: Install Ambari server using steps mentioned under below linkhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_download_the_ambari_repo_lnx6.html.Step 2: Register ambari-agent manuallyInstall ambari-agent package on all the

In Apache NiFi 1.2, there are processors for Reading Hive data via HiveQL and Storing to Hive via HiveQL. These processors are SelectHiveQL and PutHiveQL.Configuring a HiveQL processor is simple, you need to enter your query and pick either AVRO or CSV format. AVRO is a better fit, I am waiting for ORC. Most important you need to set a Connection Pool to connect to your cluster.You can just enter a regular SQL that you are doing in Hive.For Hive to work, you must setup a HiveConnectionPool Controller Service. After configuration you will need to enable that and then you can enable your processor(s). For connecting to Hive on the Sandbox, set the Database Connection URL: jdbc://hive2://localhost:10000/default. For Hive Configuration Resources: you set the hive configuration files. You can set the Database User and Password of the user that has access you require for Hive. For documentation on the HiveConnectionPool.For a PutHiveQL, you just need to set a connection po

How to delete Data Dir from the DNs 1. Open the Namenode UI and check the status. It should be healthy i.e. no missing, corrupt or under-replicated blocks   2. Go to Ambari—Services—HDFS—Configs and change the Datanode directories (remove the datadir which is not required) and save the change Note: Please just make the config change and DON’T delete the directories or any files yet. 3. IMP: We need to be very careful here NOT to restart all the services as Ambari will ask to “Restart Required”, otherwise we will see the missing blocks in NN and have to revert the changes back. 4.Go to any one of the DN and restart the DataNode service 5. Login to the same DN from putty and run the block pool report. Run the below command for same hdfs dfsadmin -triggerBlockReport <datanode_host:ipc_port> You can get the datanode ipc port here Ambari---Services---HDFS---Configs and search for dfs.datanode.ipc.address Here is the sample command output ran on the datanode [hdfs@xxxx ~]$

Credits to @mgilman for contributing to this article: Since the introduction of NiFi 1.0.0 administrators have a greater ability to manage policy through the addition of Ranger Integration and a more granular authorization model. This article provides a guide for those looking to define and manage NiFi policies in Ranger. To learn more on configuring NiFi to use Ranger via Ambari please review the parent article HDF 2.0 - Integrating Secured NiFi with Secured Ranger for Authorization Management. Behind the scenes NiFi uses a REST based API for all user interaction; therefore resource based policies are used in Ranger to define users' level of permissions when executing calls against these REST endpoints via NiFi's UI . This allows administrators to define policies by selecting a NiFi resource/endpoint, choosing whether users have Read or Write (Modify) permissions to that resource, and selecting users who will be granted the configured permission. For example the image below sho

<provider> <role>ha</role> <name>HaProvider</name> <enabled>true</enabled> <param> <name>RESOURCEMANAGER</name> <value>maxFailoverAttempts=3;failoverSleep=1000;enabled=true</value> </param> </provider> <service> <role>RESOURCEMANAGER</role> <url><Resource manager 1 url:8088>/ws</url> <url><Resource manager 2 url:8088/ws</url> </service>

1) Enable MQTT extensionPlease reference https://github.com/apache/nifi-minifi-cpp/blob/master/README.mdcmake -DENABLE_MQTT=TRUE ..2) Common Processor PropertiesBroker URI: The URI to use to connect to the MQTT brokerSession state: Whether to start afresh or resume previous flows, default to trueClient ID: MQTT client ID to useUsername: Username to use when connecting to the brokerPassword: Password to use when connecting to the brokerKeep Alive Interval: Defines the maximum time interval between messages sent or receive, default to 60 secConnection Timeout: Maximum time interval the client will wait for the network connection to the MQTT server, default to 30 secQuality of Service: The Quality of Service(QoS) to send the message with. Accepts three values 0, 1, 2Topic: The topic to publish or subscribe the message3) PublishMQTT Processor PropertiesRetain: Retain MQTT published record in broker, default to falseMax Flow Segment Size: Maximum flow content payload segment size for the MQ

Spark Thrift Server is a service that allows JDBC and ODBC clients to run Spark SQL queries on Apache Spark. By default, Spark Thrift Server runs queries under the identity of the operating system account running the Spark Thrift Server. In a multi-user environment, queries often need to run under the identity of the end user who originated the query; this capability is called "user impersonation". This article describes the work done in HDP 2.6 to support user impersonation for the Spark Thrift Server. The feature is supported in HDP 2.5.x and later versions, for Apache Spark 1 versions 1.6.3 and later. When user impersonation is enabled, Spark Thrift Server runs queries as the submitting user. By running queries under the user account associated with the submitter, the Thrift Server can enforce user level permissions and access control lists. Associated data cached in Spark is visible only to queries from the submitting user. User impersonation enables granular access c

ISSUE: While trying to run ConsumeKafka process to consume messages from secure Kafka, it throws following error:org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:702) ~[na:na] at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:557) ~[na:na] at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:540) ~[na:na] at org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:136) ~[na:na] at org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:106) ~[na:na] at org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_0_10.onTrigger(ConsumeKafka_0_10.java:285) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.0.2.1.1.0-2.jar:1.1.0.2.1.1.0-2] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) ~[nifi-fram

1.0. OverviewMonitoring enterprise applications and platforms is key responsibility of operations teams. Hortonworks HDP platform uses Ambari for monitoring and alerting, but it is unrealistic, in an enterprise environment, for operations teams to monitor each instance’s monitoring interface in its own front-end. For instance, some operations teams have to monitor 10s or 100s of HDP clusters as well as other systems, including infrastructure related systems, third party applications and databases. Monitoring each Ambari instance’s alerting GUI would be tedious. Therefore, a central dashboard is often used to monitor a broad set of clusters, platforms and applications.SNMP (Standard Network Management Protocol) provides a mechanism to achieve this goal of monitoring a broad set of applications, platforms and infrastructure, and Ambari supports SNMP alerting. This article provides a brief overview of SNMP and a guide for how to integrate HDP platform monitoring with OpenNMS. OpenNMS is a

This article describes how to add a new custom script and execute the script through Ambari. In this article, I took a simple use case to install kerberos packages through custom scripts.1) Below is the code to install kerberos packages (install_kerberos_package.py)#!/usr/bin/env pythonfrom resource_management import Script, Execute, formatfrom ambari_commons.os_check import OSCheckfrom resource_management.core import shellfrom resource_management.core.logger import Loggerclass InstallKerberosPackage(Script): def actionexecute(self, env): config = Script.get_config() structured_output = {} cmd = self.get_install_cmd() Logger.info("Installing Kerberos Package") code, output = shell.call(cmd, sudo = True) if 0 == code: structured_output["install_kerberos_package"] = {"exit_code" : 0, "message": format("Packages installed successfully")} else: structured_output["install_kerberos_package"] = {"exit_code": code, "message": "Failed to install packages! {0}".fo

Objective This tutorial walks you through how to install and setup a local Apache NiFi Registry to integrate with Apache NiFi and start using versioned NiFi dataflows. It assumes basic experience with NiFi but little to no experience with NiFi Registry.A video version of this tutorial can be seen here: https://youtu.be/X_qhRVChjZYEnvironment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.5.0 Apache NiFi Registry 0.1.0 Note: Apache NiFi 1.5.0 is the first NiFi release to support integration with the NiFi Registry. Nifi Registry 0.1.0 is the first and currently only version of the application.Apache NiFi Registry ConfigurationRegistry Installation Download the tarball of the 0.1.0 Registry release: nifi-registry-0.1.0-bin.tar.gz Extract the tar: tar xzvf nifi-registry-0.1.0-bin.tar.gzStart the RegistryIn a terminal window, navigate to the directory where NiFi Registry was installed. Run: bin/nifi-registry.sh startOpen

In previous post we have seen how to install multi node HDP cluster using Ambari Blueprints. In this post we will see how to Automate HDP installation with Namenode HA using Ambari Blueprints..Note - For Ambari 2.6.X onwards, we will have to register VDF to register internal repository, or else Ambari will pick up latest version of HDP and use the public repos. please see below document for more information. For Ambari version less than 2.6.X, this guide will work without any modifications.Document - https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_relnotes-2.6.0.0-behavioral-changes.html.Below are simple steps to install HDP multinode cluster with Namenode HA using internal repository via Ambari Blueprints..Step 1: Install Ambari server using steps mentioned under below linkhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Installing_HDP_AMB/content/_download_the_ambari_repo_lnx6.html.Step 2: Register ambari-agent manuallyInstall

Ranger plugins send their audit event (whether access was granted or not and based on the policy) directly to the configured sink for audits, which can be HDFS, Solr or both. Ranger Audit is a highly customizable event queue system that can be tailored to suit the needs of production environments. When the plugin is enabled and no specific policy is in place for access to some object, the plugin will fall back to enforcing the standard component level Access Control Lists (ACL’s). For HDFS, that would be the user: rwx / group: rwx / other: rwx ACL’s on folders and files. Once this defaulting to component ACL’s happens, the audit events show a ‘ - ‘ in the ‘Policy ID’ column instead of a policy number. If a Ranger policy was in control of allowing/ denying, the policy number is shown. Key Things to Remember Access decisions taken by Ranger (to allow/ deny user) are based on a combination of three things: resource - that is being accessed user/group - who is trying to access opera

This article is an extension to the official HDP document.Apart from following the steps listed in this document, you must perform the following checks to ensure the hook is configured correctly and does not result in errors when you start executing queries in Hive.1. In hive-site.xml, verify hive.server2.async.exec.threads is not set to 1. If so, then increase to 100.2. In hive-site.xml, verify the max thread pool size is not set to 1. Increase it to 5 to begin with and you may need to increase it further depending on the load.Recommended values: <property> <name>atlas.hook.hive.maxThreads</name> <value>5</value></property><property> <name>hive.server2.async.exec.threads</name> <value>100</value></property>

Summary Enabling SSL encryption for the Web UIs that make up Hadoop is a tedious process that requires planning, learning to use security tools, and lots of mouse clicks through Ambari's UI. This article aims to simplify the process by presenting a semi-automated, start-to-finish example that enables SSL for the below Web UIs in the Hortonworks Sandbox: Ambari HBase Oozie Ranger HDFSPlanning There is no substitute for reading the documentation. If you plan on enabling SSL in a production cluster, then make sure you are familiar with SSL concepts and the communication paths between each HDP component. In addition, plan on cluster downtime. Here are some concepts that you should know well: Certificate Authority (CA) A Certificate Authority is a company that others trust that signs certificates for a fee. On a Mac you can view a list of CAs that your computer trusts by opening up the "Keychain Access" application and clicking on "System Roots". If you don't want to pay one of

WARNING: this is a workaround and does not mean is a certifiable solution from Hortonworks. In certain scenarios, customers are required to run client and services on different OS versions and flavors. I will only cover clients for Pig and Hive, certainly this procedure can be applied to services but it's more involved. It is advised that you contact Hortonworks support if you go down this path.Setup scenario:3 node Ubuntu 12.04 cluster with HDP 2.4.2 and Ambari 2.2.2.01 node Centos 6 and no Ambari.on the centos6 node, download and install Java, preferably the same as on other nodes. I followed this document to install Oracle JDK 8 as that's what is running on my Ubuntu cluster. https://www.digitalocean.com/community/tutorials/how-to-install-java-on-centos-and-fedoraon the centos6 node, download the hdp repowget http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0/hdp.repocp hdp.repo /etc/yum.repos.dyum install hadoop pig hive-hcatalog hive-webhcat tezcopy the /etc/hado

In an Apache NiFi cluster, every node runs the same dataflow and data is divided between the nodes. In order to leverage the full processing power of the cluster, the next logical question is - "how do I distribute data across the cluster?". The answer depends on the source of data. Generally, there are sources that can push data, and sources that provide data to be pulled. This post will describe some of the common patterns for dealing with these scenarios. Background A NiFi cluster is made up of a NiFi Cluster Manager (NCM) and one or more nodes. The NCM does not perform any processing of data, but manages the cluster and provides the single point of access to the UI for the cluster. In addition, one of the processing nodes can be designated as a Primary Node. Processors can then be schedule to run on the Primary Node only, via an option on the scheduling tab of the processor which is only available in a cluster. When connecting two NiFi instances, the connection is made with

Periodically, and when there are no regions in transition, a load balancer will run and move regions around to balance the cluster’s load. See Balancer for configuring this property. The Load Balancer ensures that the region replicas are not co-hosted in the same region servers and also in the same rack (if possible).The HDFS balancer attempts to spread HDFS blocks evenly among DataNodes. HBase relies on compactions to restore locality after a region split or failure. These two types of balancing do not work well together.In the past, the generally accepted advice was to turn off the HDFS load balancer and rely on the HBase balancer, since the HDFS balancer would degrade locality. HDFS-6133 provides the ability to exclude favored-nodes (pinned) blocks from the HDFS load balancer, by setting thedfs.datanode.block-pinning.enabled property to true in the HDFS service configuration.HBase can be enabled to use the HDFS favored-nodes feature by switching the HBase balancer class (conf: hbase