Community Articles

Introduction OpenID Connect (OIDC) is an authentication layer on top of OAuth 2.0, an authorization framework. It uses simple JSON Web Tokens (JWT), which is an open standard for securely transmitting information as a JSON object. These objects are normally signed with an RSA key and contain information such as if the user was authenticated or the user's id and email. More information and examples can be found here: https://auth0.com/docs/tokens/concepts/jwts Knox, together with pac4j, a Java security engine, uses OpenID Connect (and also SAML, CAS, OAuth) to enable Single Sign On using a variety of 3rd party Identity Providers that support these protocols. For example, Knox can be integrated with a service such as https://auth0.com to provide users authenticated to auth0 access to Hadoop resources (without a need to enter their credentials again or even integrate Knox with an LDAP). Following is an example of how Knox can integrate with auth0 using OpenID Connect. Setup auth0 Sign up

Objective This tutorial walks you through how to install and secure a NiFi Registry using client certificates. A quick example of modifying user privileges in the Registry is also included.A video version of this tutorial can be seen here: https://youtu.be/qD03ao3R-a4 Note: To learn the basics of setting up an unsecured Registry and integrating with Apache NiFi see the HCC article Versioned DataFlows with Apache NiFi 1.5 and Apache NiFi Registry 0.1.0.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi Registry 0.1.0 Apache NiFi Toolkit 1.5.0Secure NiFi Registry ConfigurationDownload & Extract Tarballs Download the tarball for the 0.1.0 Registry release: nifi-registry-0.1.0-bin.tar.gz and the tarball for the 1.5.0 NiFi Toolkit: nifi-toolkit-1.5.0-bin.tar.gz Extract the tars: tar xzvf nifi-registry-0.1.0-bin.tar.gz tar xzvf nifi-toolkit-1.5.0-bin.tar.gzGenerate Configuration and Certificate Files We will use t

We can configure Hive View 2.0 to view Ranger Policies associated with any Table. It will not be configured by default (as of HDP 2.6.3) and we need to follow the below steps:1. Go to Ambari Dasboard -> admin -> Manage Ambari2. In the Manage Ambari page under Views section, expand Hive and then click on Hive View 2.03. Under the setting section, please provide the details as below:- Ranger Service Name: Go to Ranger Admin UI and check the Ranger Service Name which is configured for Hive (shown as below). Here it is tsk_hive- Provide the Ranger Admin username and password as you have configured- WebHDFS Authentication for kerberos enabled cluster: auth=KERBEROS;proxyuser=<proxyuser> Provide the ambari user principal name in place of <proxyuser>4. After adding the above details, click on "Save". Leave the other settings as default.Additional Steps (only when you are not using local cluster): - If you are using the Local Cluster, then it is not required to change any pa

Whenever in SmartSense bundle if ranger metric related information is not there and if there is any consistent error or if the json files are completely blank. There is probability that the metric generating scripts doesn't get environment variable so please perform the below given steps and verify again.1) Open file /etc/ambari-server/conf/ambari.properties 2) Check for java.home variable and copy its value 3) Go to Ranger conf directory, cd /usr/hdp/current/ranger-admin/conf/ 4) Create a new file ranger-admin-env.sh with below content and save it.export JAVA_HOME=<paste the java.home variable path value copied from ambari.properties file> 5) Go to Ranger KMS conf directory, cd /usr/hdp/current/ranger-kms/conf/6) Create a new file ranger-kms-env.sh with below content and save it.export JAVA_HOME=<paste the java.home variable path value copied from ambari.properties file> 7) From Amabari Restart Ranger and Ranger KMS😎 Generate bundle from Ambari -> SmartSense View an

The intent of this article is to show how NiFi policies in Ranger map to what you would see when using NiFi's default file based authorizer via the NiFi UI. This article will cover what access each of the policies granted to the entities (user and server) that assigned to them.There are controller level policies and component level policies in NiFi.The controller level policies are not tied to any specific component uuid. In Ranger those policies will just show as/<some policy name>-These include the following: Ranger Policy (Base policies):NiFi Policies (Hamburger menu)Ranger permissions description:/resources *** Note: No policies will be available until this policy is manually added.N/AThis policy allows Ranger to retrieve a listing of all available policies from NiFi. The server/user from the keystore being used by Ranger must be granted “read” privileges to this resource./flow* See note [3] belowView the user interfaceRead/View - This policy gives users the ability to v

Prerequisite 1. You have HDF-1.2 installed on your server2. Make sure KDC is installed on your server and is started, will try to describe steps briefly in the tutorial, below is the link to detailed steps from HDP documentation:HDP-Documenation-for-Kerberos Installing and Configuring KDC:1. Lets install a new version of KDC server:# yum -y install krb5-server krb5-libs krb5-auth-dialog krb5-workstation2. Using a text editor, open the KDC server configuration file, located by default here:# vi /etc/krb5.conf [realms] EXAMPLE.COM = { kdc = node1 admin_server = node1 }add your host name, mine is node1.3. Use the utility kdb5_util to create the Kerberos database, when asked lets put password ‘hadoop’:# kdb5_util create –s 4. Lets Start the KDC server and the KDC admin server, set them to auto-start on boot# /etc/rc.d/init.d/krb5kdc start# /etc/rc.d/init.d/kadmin start# chkconfig krb5kdc on# chkconfig kadmin on5. Lets add a service principal for a server and export the keytab from t

The solution in this article was tested for DB2 and could work for other databases as well , that are not available in the GenerateTableFetch Database Type dropdown.I was trying to develop a flow that extracted data from a few tables in DB2 database and the goal was to land that data into HDFS. Our flow was a Basic ListDatabaseTables followed by a GenerateTableFetch followed by a ExecuteSSQL. The idea was to generate multiple queries that could parallelly extract data from the db2 nodes, utilizing all the 3 NiFi nodes in the cluster.The problem we ran into was that there is no option for DB2 in the GenerateTableFetch DBType property. Which means we had to select the generic option.Upon running the flow, the select query failed to execute on DB2. On investigating we found that the query generated by GenerateTableFetch looked like thisselect id,name,update_dt,state,city,address from customers where update_dt<='01-01-2018 12:00:00' order by update_dt offset 10000 limit 10000The limit

Here are the steps to get pyspark working on SHCa) add the following into Ambari -> Spark -> Configs -> Advanced spark-env -> spark-env templateexport SPARK_CLASSPATH=/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.6.0.3-8/spark/conf/b) kinit as e.g. hbase userc) run$ pyspark --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/d) call each line separately:from pyspark.sql import Row data = range(0,255) rdd = sc.parallelize(data).map(lambda i : Row(name=i,age=i)) import json cat = json.dumps({"table":{"namespace":"default", "name":"dk", "tableCoder":"PrimitiveType"},"rowkey":"key","columns":{"name":{"cf":"rowkey", "col":"key", "type"

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. I find myself quite often puzzled when required to provide differences between releases and just reading the release notes history at https://cwiki.apache.org/confluence/display/NIFI/Release+Notes and looking at the latest list of NiFi processors is not trivial to determine which new processors were added.I put together matrix which I hope will help developers to take advantage of new processor to improve old and develop new flows.In a nutshell, main functionality added is around:AzureEventHubKafka 0.11 and 1.0 processorsRecord processorsRethinkDBFlatten JsonExecute Spark InteractiveExecute Groovy ScriptMy favorite improvements are around record processors, flattening JSON and executing Spark interactively.The following is a table of the matrix, arranged alphabetically from A-D:See here for the Matrix Table from E-JSee here for the Matrix Tabke from K-ZForNiFi 1.5NiFi 1.4NiFi 1.3N

In this article, we will see how an organization can tackle the data science challenge for an Internet of things use case. This article shows how a trucking company can leverage data and analytics to solve its logistics problem by leveraging Hortonworks Data Platform (HDP), Apache NiFi, and IBM’s Data Science Experience (DSX) to build a real-time data science solution to predict any violation events by the driver. IBM DSX is a data science platform from IBM that provides all favorite data science toolkits at one place. DSX and HDP solve some of the major pain points of running data science at scale such as - ease of deployment and maintenance of machine learning models, scaling the developed model to big data scale, ease of using different open source tools, and collaboration. DSX comes in multiple flavors: cloud, desktop, and local. DSX can run on top of you HDP cluster to provide advanced analytics capabilities. Following diagram show major stages of a data science project -Stage 1:

To print GC details, please add the following line in Spark--> config--> Advanced spark-env --> spark-env template and restart spark history server.export SPARK_DAEMON_JAVA_OPTS=" -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:{{spark_log_dir}}/spark_history_server.gc.`date +'%Y%m%d%H%M'`"

Rolling upgrade from HDF-3.0.2.0 to 3.1In HDF 3.1 the concept of Rolling Upgrade will be introduced for the first time. Prior to this release, HDF only supported Express Upgrade which orchestrated a stop all, upgrade and start all approach to upgrades.Rolling Upgrade orchestrates the upgrade in an order that is meant to preserve cluster operation and minimize service impact during upgrade. This process has more stringent prerequisites (particularly regarding cluster high availability configuration) and can take longer to complete than an Express Upgrade.In the case of HDF, we have components that do not need to be brought down while upgrading HDF from older to newer version. These include Storm (long running topologies), Kafka (data being ingested into topics or topics being accessed by other services), Streaming Analytics Manager (Live streaming analytics applications), Schema Registry (shared repository of schemas to allow application to interact with each other) t Therefore from

In this article I will explain an easy way to automate some basic tasks in NiFi as an introduction to NiPyApi, an automation package for Apache NiFi and its sub-projects.Environment Setup RequirementsYou will need a Python environment 2.7 or 3.6 are tested, and most computers come with one of theseor you can create a virtualenvor you can install Python on OSX using homebrewYou will need a recent internet browser, given you're reading this I assume you have one - I'm using ChromeYou will need NiFi services to test against - if you have Docker installed you can use the following commands to prepare NiFi & NiFi Registry services:user$: curl https://raw.githubusercontent.com/Chaffelson/nipyapi/master/test_env_config/docker_compose_latest/docker-compose.yml | docker-compose -f - up -duser$: docker psDocker will download and start the NiFi containers, and show you the details:You should be able to browse to both NiFi and NiFi-Registry at the following URLs:http://localhost:8080/nifi/htt

Problem:While implementing Auto-Hdfs, following errors were thrown in Nimbus log:Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: https://da0gdal202.match.corp:9393/kms/v1/?op=GETDELEGATIONTOKEN&doAs=gdsreader&renewer=hdfs-hdpprod%40MATCH.CORP&user.name=hdfs, status: 403, message: Forbiddenat org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)at org.apache.hadoop.security.au

Assuming you start with a kerberized HDP cluster with Hbase installed.First check what your service principal is i.e.klist -kt /etc/security/keytabs/hbase.service.keytabKeytab name: FILE:hbase.service.keytabKVNO Timestamp Principal---- ----------------- -------------------------------------------------------- 2 12/20/16 13:51:21 hbase/hdp252.hdp@HWX.COM 2 12/20/16 13:51:21 hbase/hdp252.hdp@HWX.COM 2 12/20/16 13:51:21 hbase/hdp252.hdp@HWX.COM 2 12/20/16 13:51:21 hbase/hdp252.hdp@HWX.COM 2 12/20/16 13:51:21 hbase/hdp252.hdp@HWX.COM In Ambari head to Hbase -> Configs -> Advanced -> Custom Hbase-Site.xml and add the following new parameters with the keytab / principal substituted: hbase.thrift.security.qop=authhbase.thrift.support.proxyuser=truehbase.regionserver.thrift.http=truehbase.thrift.keytab.file=/etc/security/keytabs/hbase.service.keytab hbase.thrift.kerberos.principal=hbase/_HOST@HWX.COM hbase.security.authentication.spnego.kerberos.keytab=/etc/security/

First, you should go to the Apache Spark downloads web page to download Spark 2.0. Link to Spark downloads page:http://spark.apache.org/downloads.htmlSet your download options (shown in image below), and click on the link next to "Download Spark" (i.e. "spark-2.0.0-bin-hadoop2.7.tgz"):This will download the gzipped tarball to your computer.Next, startup the HDP 2.5 Sandbox image within your virtual machine (either using VirtualBox or VMFusion). Once the image is booted, startup a Terminal session on your laptop and copy the tarball to the VM. Here is an example using the 'scp' (secure copy) command, although you can use any file copy mechanism.scp -p 2222 spark-2.0.0-bin-hadoop2.7.tgz root@127.0.0.1:~ This will copy the file to the 'root' user's home directory on the VM. Next, login (via ssh) to the VM:ssh -p 2222 root@127.0.0.1Once logged in, unzip the tarball with this command:tar -xvzf spark-2.0.0-bin-hadoop2.7.tgzYou can now navigate to the "seed" directory already created for Spar

Note:Ranger communicates with Plug-ins only with 2 WAY SSL (1 way SSL in not allowed).[Updated] Appears like one way SSL is possible with latest patch - https://issues.apache.org/jira/browse/RANGER-1094First get server keystore as skeystore.jks and truststore strustore.jks and client keystore as ckeystore.jks and ctruststore.jks (you can create these keystore/truststores once you get the signed certs from CA Signing.here is the steps:1. Login to AmbariGo to Ranger > Configs > Ranger Settings > External URL points to a URL that uses SSL: https://<hostname of Ranger>:<https port, default is 6182> andranger.service.https.attrib.ssl.enabled to false 2. Go to HDFS > Configs > Advanced > ranger-hdfs-policymgr-ssl and set the following properties: xasecure.policymgr.clientssl.keystore = /etc/hadoop/conf/ckeystore.jks xasecure.policymgr.clientssl.keystore.password = bigdata xasecure.policymgr.clientssl.truststore = strustore.jks xasecure.policymgr.clie

There are two sections which illustrate moving the Journal directory on the same Host. You can follow one of the section according to your need. Section 1: ( Service Downtime is not required ) 1. Change the settings in Ambari for journal node edits directory(dfs.journalnode.edits.dir) from /hadoop/hdfs/journal/ to /data/1/journal/ 2. Don't restart any services immediately. 3. Stop the journal node on NODE1. a. SSH to NODE1b. sudo mkdir -p /data/1/journalc. sudo chown hdfs:hadoop /data/1/journald. sudo rsync -ahvAX /hadoop/hdfs/journal/* /data/1/journal/ Start the journal node on NODE1 4. Repeat step 3 for remaining two journal nodes NODE2 and NODE3. 5. Restart the required services accordingly (Rolling or All at once) Section 2: ( Service Downtime is required ) 1. Create the new directory on all Journal nodes. Example:# mkdir -p /data/1/journal/ 2. Change ownership to "hdfs:hadoop" on all Journal nodes. Example:# chown hdfs:had

After trying to disable kerberos via the UI the following message is seen in the ambari-server logs and the service fails to start....INFO [main] KerberosChecker:57 - Checking Ambari Server Kerberos credentials. WARN [main] KerberosChecker:81 - /etc/security/keytabs/ambari.server.keytab doesn't exist ERROR [main] KerberosChecker:115 - Configuration Error - either doNotPrompt should be false or at least one of useTicketCache, useKeyTab, tryFirstPass and useFirstPass should be true Edit: /etc/ambari-server/conf/ambari.properties Ensure the following is set: kerberos.check.jaas.configuration=false Further disable kerberos for ambari server by removing: "-Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf-Djava.security.krb5.conf=/etc/krb5.conf-Djavax.security.auth.useSubjectCredsOnly=false" from AMBARI_JVM_ARGS value in /var/lib/ambari-server/ambari-env.sh If you decide to enable kerberos again, you have to manually revert changes in /var/lib/ambari-serv

Short description: This article describes how to enable Knox SSO authentication on Kerberos enabled Oozie UI The goal is to provide a single authentication handler for redirecting to an external endpoint for acquiring a JWT (JSONWebToken) token to be used as a SSO representation of an authentication event. The hadoop common JWTRedirectAuthenticationHandler can be used to add this support to component UIs. It is an extension of AltKerberosAuthenticationHandler, which require kerberos to be enabled on the HTTP endpoints in order to work. This means that UIs will use SSO for authentication, but at the same time REST APIs will still be using kerberos. For authenticating REST APIs with Knox SSO as well, there is a filter based authentication provider in Knox. SSL must be enabled on Oozie server to be able to use redirection. Enable SSL on Oozie Configuration in Ambari Set the following properties at the Oozie config page: oozie.authentication.type=org.apache.hadoop.s

Objective This tutorial demonstrates how to use the PutMongoRecord processor to easily put data from a CSV file into MongoDB. Note: The PutMongoRecord processor was introduced in NiFi 1.4.0. As such, the tutorial needs to be done running Version 1.4.0.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.4.0 MongoDB 3.4.9PutMongoRecord (CSVReader)Demo ConfigurationMongoDB For my environment, I had a local MongoDB 3.4.9 instance installed. Start MongoDB and create a database "hcc" and a collection "movies" for use with this tutorial: >use hcc switched to db hcc > db.createCollection("movies") { "ok" : 1 } > show collections movies I like to use Robot 3T to manage/monitor my MongoDB instance:Initial Flow A powerful feature about the record-oriented functionality in NiFi is the ability to re-use Record Readers and Writers. In conjunction with the Record processors, it is quick and easy to change

There are situations, when one might want to submit a Spark job via a REST API: If you want to submit Spark jobs from your IDE on our workstation outside the cluster If the cluster can only be accessed via Knox (perimeter security) One possibility is to use the Oozie REST API and the Oozie Spark action, However, this article looks into the option of using the YARN REST API directly. Starting with the Cluster Applications API I tried to come up with an approach that resembles the spark-submit command. 1) Copy Spark assembly jar to HDFS Per default the spark assembly jar file is not available in HDFS. For remote access we will need it. Some standard locations in HDP are: HDP 2.3.2: Version: 2.3.2.0-2950 Spark Jar: /usr/hdp/2.3.2.0-2950/spark/lib/spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar HDP 2.4.0: Version: 2.4.0.0-169 Spark Jar: /usr/hdp/2.4.0.0-169/spark/lib/spark-assembly-1.6.0.2.4.0.0-169-hadoop2.7.1.2.4.0.0-169.jar This is a one

Ambari is a great tool to manage the Hortonworks Data Platform. However, as the complexity of the applications that run on HDP grows and more and more components from the stack are required, it may be necessary to begin automating tasks like configuration changes. Using the Ambari REST interface it is possible to install, control, interrogate, and even change configuration of HDP services. This example demonstrates how to automate the install and configuration of Nifi. However, the same approach can be applied to any service. This example uses curl to make all REST API request.Make sure that the bits required to actually install the service in Ambari are available on the host running the Ambari server. In the case of Nifi, the bits should be present in /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI.1. Check Service Status:curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFIUse regex of so

' The Onion Omega 2+ is a small IoT device that runs a simple busybox Linux and can run Micropython. This let's you run some simple application and interact with some sensors and an OLED. Onion Omega 2+ Stats 580MHz Cpu128MB memory32mb storageadded usb 32gbusb 2microusbb/g/n wifi15 gpio2 pwm2 uart1 i2c1 spi1 i2s Setting Up the Omega opkg install python-pippip install --upgrade setuptoolspip install paho-mqttopkg install pyOledExpUpgrading pyOledExp on root from 0.4-1 to 0.5-1...Downloading http://repo.onion.io/omega2/packages/onion/pyOledExp_0.5-1_mipsel_24kc.ipkConfiguring pyOledExp.mkdir /mnt/sda1mount /dev/sda1 /mnt/sda1./run.sh> Initializing display> Setting display to ON> Enabling horizontal scrolling to the left> Writing '[{"ipaddress": "192.168.1.176", "endtime": "2018-01-29 00:50:39", "end": "1517187039.44"}]' to display0crontab -ecrontab -l*/1 * * * * /opt/demo/run.sh1517187305: New connection from 192.168.1.176 on port 1883.1517187305: New client conn

The Matrix Creator is an interesting multiple sensor hat that fits on a Raspberry Pi 3.First step is to connect it, which is a simple snap, no soldering required.The specs are pretty impressive:Xilinx Spartan 6 XC6SLX4 FPGAAmtel Cortex-M3 ATSAM3S2 Microcontroller8 MEMS MP34DB02 audio sensor digital microphonesST LSM9DS1 3D accelerometer, 3D gyroscope, 3d magnetometer IMUST HTS221 capacitive digital sensor for relative humidity and temperatureNXP MPL3115A2 precision pressure sensor with altimetrySilicon Labs EM358X - 2.4 GHz IEEE 802.15.4 ZigbeeSigma Designs ZM5202 - 868/908/921 MHz ZWaveVishay TSOP573 - carrier 38.0 kHz IR ReceiverVishay VEML6070 UV light sensorNXP PN512 NFC readerEverloop 35 RGBW LEDSIt runs on Raspian lite and installs via:curl https://matrix-io.github.io/matrix-documentation/install.sh | shOur Apache NiFi Flow For Processing the Three Types of DataOur Versioned Apache NiFi and MiniFi FlowsWe tail the three files produced by the three example Python sensors readersBo