Community Articles

When synchronizing users from AD or LDAP to Ranger using usersync, It is possible that there are spaces in usernames. These spaces in user names and Group Names may be replaced by SSSD to use underscores. (Example SSSD config section below. [sssd]override_space = _ Ranger needs to synchronize users similarly to avoid issues in access permission setup. To enable Ranger to perform this conversion , the following flags can be added to "Ranger --> Configs --> Advanced --> custom ranger-ug-site" By adding 2 propertiesranger.usersync.mapping.groupname.regex, ranger.usersync.mapping.username.regex These two properties can be set to s/ /_/gAfter setting these properties restart ranger service using Ambari and then restart Ranger user sync. User sync will add new user names with Underscores in the user name. Older usernames and group names with Spaces will need manual cleanup. Note: This f

MiNiFi C++ implemented the secure site to site raw socket transport using OpenSSL/TLS.PR MINIFI-184: Add Security Support https://github.com/apache/nifi-minifi-cpp/commit/63dbb8241e851068bff54ab8cef8310cc4a22cb5 implemented the sameIt is using both client and server certificate to do mutual authentication between client and server via OpenSSL/TLS.Please look at http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site about setting up the NiFi server site security configuration. Before you start to configure MiNiFi C++, you need to have client certificate PEM file, client private key PEM file, client CA certificate PEM file, passphrase to the client certificate. Client certificate PEM file, client private key PEM file can be combined into a single PEM file.You can use openssl tool to convert different certificates formats between PEM/PKCS, etc.Modify conf/minifi.properties to add Site to Site secure setting### Site2Site Security Configuration in minifi.prop

ProblemI have my shiro configured as[users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections #admin = password1 admin = change #user1 = password2, role1, role2 #user2 = password3, role3 #user3 = password4, role2 # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = ou=my,ou=company,o=com ldapRealm.userDnTemplate = uid={0},ou=my,ou=company,o=com ldapRealm.contextFactory.url = ldap://<ldap_host>:389 ldapRealm.contextFactory.authenticationMechanism = SIMPLE # doc horton ldapRealm.contextFactory.systemUsername = <system_username_bind>ldapRealm.contextFactory.systemPassword = <system_username_password> #sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSess

Sometime we get a situation where we have to get lists of all long running and based on threshold we need to kill them.Also sometime we need to do it for a specific yarn queue. In such situation following script will help you to do your job.#!/bin/bashif [ "$#" -lt 1 ]; then echo "Usage: $0 <max_life_in_mins>" exit 1fiyarn application -list 2>/dev/null | grep "report" | grep "RUNNING" | awk '{print $1}' > job_list.txtfor jobId in `cat job_list.txt`dofinish_time=`yarn application -status $jobId 2>/dev/null | grep "Finish-Time" | awk '{print $NF}'`if [ $finish_time -ne 0 ]; then echo "App $jobId is not running" exit 1fitime_diff=`date +%s`-`yarn application -status $jobId 2>/dev/null | grep "Start-Time" | awk '{print $NF}' | sed 's!$!/1000!'`time_diff_in_mins=`echo "("$time_diff")/60" | bc`echo "App $jobId is running for $time_diff_in_mins min(s)"if [ $time_diff_in_mins -gt $1 ]; then echo "Killing app $jobId" yarn application -kill $jobIdelse echo "App $job

In the big data and distributed system world you may have done a world class job in dev and Unit testing, chances are you still did it on sample data moreover in dev you might not have done it in a truly distributed system but more on one machine simulating distribution. Hence on the prod cluster you could rely on logs and sometime you would also like to connect a remote debugger and other tools you are used to. In this blog post I will go over an example in Java and Eclipse. Partly because I use those and also because I see mostly scala examples so let’s give java a little love.Setting upI have installed a Hortonworks sandbox for this exercise, you can donwload it here. You will also need to open up a port to bind on, on the last post I used port 7777 i’ll stick with this here as well.First we will use a very standard wordcount example you can get in all the spark tutorials. Set up a quick maven project to bundle your jar and dependent libs you might be using. Notice the breakpoint on

Caveat: This feature has been validated by HWX engineer manually but we don't officially support it at the moment.Environment: HDP-2.5.3.0-37Ambari-2.4.2.0-136JDK1.8Kerberos enabledRanger enabledDue to security limitations, we can only launch flume agent processes in Ambari.STEP 1: Create/modify flume configuration file.Ambari -> Flume -> Configs -> flume.conf# Flume agent config#### Global ####demo.sources = logtcp logudpdemo.channels = kafka_channeldemo.sinks = sink#### Sources ####demo.sources.logtcp.type = multiport_syslogtcpdemo.sources.logtcp.ports = 9515demo.sources.logtcp.host = 0.0.0.0demo.sources.logtcp.keepFields = truedemo.sources.logtcp.selector.type=replicatingdemo.sources.logtcp.channels= kafka_channeldemo.sources.logudp.type = syslogudpdemo.sources.logudp.port = 9515demo.sources.logudp.host = 0.0.0.0demo.sources.logudp.keepFields = truedemo.sources.logudp.selector.type=replicatingdemo.sources.logudp.channels = kafka_channel#### Sinks ####demo.sinks.sink.type =

Problem When running$ ambari-server restartI can see "tones" of these messages in ambari-server.log27 Feb 2017 11:38:00,942 WARN [main] AbstractLifeCycle:204 - FAILED SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already in usejava.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.j

Objective This tutorial is designed to walk you through the process of creating a MiniFi flow to read data from a Sense HAT sensor on a Raspberry Pi 3. The MiniFi flow will push data to a remote NiFi instance running on your computer. The NiFi instance will push the data to Solr. While there are other tutorials and examples of using NiFi/MiniFi with a Raspberry Pi, most of those tutorials tend to use a more complicated sensor implementation. The Sense HAT is very easy to install and use.Prerequisites You should have a Raspberry Pi 3 Model B: Raspberry Pi 3 Model BI recommend a 16+GB SD card for your Raspberry Pi 3.Don't forget to expand the filesystem after the OS is installed: raspi-config You should have a Sense HAT: Sense HATYou should already have installed the Sense HAT on your Raspberry Pi 3. You should already have installed Raspbian Jessie Lite on your Raspberry Pi 3 SD card: Raspbian Jessie LiteThe instructions for installing a Raspberry Pi OS can be found here:

This also includes an analysis on the fly for showing odds on a Craps game. This example shows a simple use of Nifi - HDF - handling multiple streams of Dice data - each one simulating a separate Craps table – showing a Monte-Carlo simulation and results of a 1000 run – emulating throws each second.To demonstrate this capability we generate some random dice data, each stream generation uses independent thread. We throttle the threads to sleep for a second between throws, mainly to demonstrate an ongoing stream of data over time.Source for data generation:https://github.com/eorgad/Dice-nifi-streams-example/tree/master/Dice-nifi-stream-example/Dice-nifi-streams/srcWe use Nifi to create a streaming flow of that data as it is being generated.This simulation will use the following Nifi processors:HandleHttpRequest (Starts an HTTP Server and listens for HTTP Requests)RouteOnAttribute (Routes FlowFiles based on their Attributes using the Attribute Expression Language)ExecuteStreamCommand (Exe

Sometime you may have a requirement where you need last accessed time of file in hdfs, then you may get it via following ways:Option 1: Ranger Audit: You can get it via ranger audit but I would not prefer it because output of audit file would be confusing due to many tmp files and dirs which present in audit db.So now to solve this requirement and to fulfill my purpose I have used java APIs and done my work very efficient and in a vary cleaned way. Option 2: Use java and build a small program. With the help of java program you can get some other useful information about hdfs file also in few number of lines. Step 1 : Create java program :package com.saurabh;import java.io.*;import java.util.*;import java.net.*;import java.nio.file.Files;import java.nio.file.Paths;import org.apache.hadoop.fs.*;import org.apache.hadoop.conf.*;import org.apache.hadoop.io.*;import org.apache.hadoop.mapred.*;import org.apache.hadoop.util.*;// For Date Conversion from long to human readable.import java.text.

HDF/NiFi is a key element in the architectural transformations with the future of data. It is a great tool to manage data flows but it also provides means to access data in motion as a stream source. This feature gives an easy way to expose data endpoints in data flows allowing users to access the data as soon as possible and perform desired analytics, online machine learning, etc. Use case description In this article, I will describe how to easily implement the following scenario: Use Apache NiFi to gather tweets filtered on specifics terms, Use Apache Spark streaming to retrieve the data, transform it and store it into an Apache Hive table, Use Apache Zeppelin to access the data and display real-time analytics. This example will use the US Presidential Election as a subject and will show: The frequency of tweets mentioning twitter accounts of one of the main candidates: Ted Cruz, Donald Trump, Hillary Clinton and Bernie Sanders. A sentiment analysis of the tweets with the pe

Steps to enable Trace logging in Ambari server for Oracle Database.Step1:1. Download ojdbc6_g.jar from Oracle JDBC driver trace jar and copy to /usr/share/java and make sure to have correct permissions. Note: take out the actual ojdbc.jar from the location.2.Create /etc/ambari-server/conf/oracle.properties.level=ALLoracle.jdbc.level=INFOoracle.jdbc.handlers=java.util.logging.ConsoleHandlerjava.util.logging.ConsoleHandler.level=INFOjava.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter3. Add "-Doracle.jdbc.Trace=true -Djava.util.logging.config.file=/etc/ambari-server/conf/oracle.properties" property in /var/lib/ambari-server/ambari-env.shexport AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djavto export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Doracle.jdbc.Trace=true -Djava.util.logging.config.file=/etc/ambari-server/con

Please find the attached "Storm/Kafka Best Practices Guide". This document is intended to be an unofficial guide to developing and deploying streaming applications using Storm and Kafka. This is not the official Hortonworks documentation, but meant to be a collection of best practices from teams implementing Storm and Kafka in Production. Pre-ReadingIf you are new to Storm Development or operations, please review the Tutorial under the Storm Documentation “Read these first” section. This document covers the basics of Storm components and operations.DevelopmentThe Storm developer has much opportunity to make the topology fast, efficient, and easier to run in Production by his Operations team. This section will discuss the best practices from the developer’s perspective to achieve these goals.Developing for highest PerformanceKeep the Tuple processing code as light as possible. Code so that the execute() method in your Bolts can run as quickly as possible. This will have multiplier effec

Introduction SQL development tools like DbVisualizer, SQuirreL SQL and DataGrip are popular options for database development. Although these tools don't offer native Hive support they can be easily configured to connect to Hive using JDBC. While connecting these tools to clusters without kerberos is relatively straightforward, the process of connecting them to kerberized clusters can be complex and error prone. This article, in combination with a project I created ( Hive JDBC Uber Jar), aim to simplify and standardize this process.Prerequisites There are a few key things that must be properly configured before attempting to connect to a kerberized cluster. A full description of these tasks is out of scope for this article, but at a high level, make sure that: You have downloaded the latest release of my Hive JDBC Uber Jar and placed it somewhere sensible DbVisualizer and/or DataGrip have been successfully installed on your workstation The krb5.conf file on your workstatio

One of the earliest improvement requests for the Apache NiFi project was to allow a user the ability to easily align components on the canvas. Well, in the upcoming 1.2 release it's HERE! In this article I will demonstrate how to align multiple selected components either vertically or horizontally.Previously, when building a flow a user would drop components on the canvas into a layout. Typically users will attempt to organize their data flow into rows and columns. This was a tedious task since aligning these components was a manual effort of dragging and dropping components to line them up 'just right'. Let's look at a crude example of such a layout:Let's see how we can organize this flow with the new 'click to align' features. First, let's align the rows:Ok great! Now let's align the columns:BOOM! And there you have it! A nicely aligned layout for an organized data flow. I hope you have found this tutorial useful and I look forward to your feedback.

NiFi works with FlowFiles. Every FlowFile that exists consists of two parts, FlowFile content and FlowFile Attributes. While the FlowFile's content lives on disk in the content repository, NiFi holds the "majority" of the FlowFile attribute data in the configured JVM heap memory space. I say "majority" because NiFi does swapping of Attributes to disk on any queue that contains over 20,000 FlowFiles (default, but can be changed in the nifi.properties).Once your NiFi is reporting OutOfMemory (OOM) Errors, there is no corrective action other then restarting NiFi. If changes are not made to your NiFi or dataflow, you are surely going to encounter this issue again and again.The default configuration for JVM heap in NiFi is only 512 MB. This value is set in the nifi-bootstrap.conf file.# JVM memory settingsjava.arg.2=-Xms512mjava.arg.3=-Xmx512mWhile the default may work for some dataflow, they are going to be undersized for others. Simply increasing these values till you stop seeing (OOM

Authentification 1- Kerberos : Kerberos is mandatory for prod environments, you can either use your AD embeded kerberos or install a new dedicated KDC - Kerberos must be in HARisk not doing the above : User impersonation for the services accounts ( jobs can be exported to run as super user permission )2 - Use a firewall to block all inbound traffic to the cluster- all sources / all ports except from the edge node ( Gateway )Risk not doing the above : Passwords in the wrong hands will systematically give access to the cluster3- Check the permissions of the keytabs, detailed this article here : Script to fix permissions and ownership of hadoop keytabs Risk not doing the above: utilisation by other cluster users 4- Use Knox for all API calls to the cluster.Benefits : inbound from "trusted" known machines, that requires authentification against an existing LDAP.Network 1 - The cluster must be in an isolated subnet - no interference with other networks, for security and thouroughputR

In order to setup HTTPs access for LogSearch Portal UI first of all we will need to create the keystore. We will also create truststore as well using the "$JAVA_HOME/bin/keytool" utility. In this example we will be using the self signed certificates.# mkdir -p /etc/security/serverKeys/# keytool -genkey -v -alias logsearch -keyalg RSA -keysize 1024 -keystore /etc/security/serverKeys/logsearch.keyStore.jks -validity 3650 -keypass logsearch -storepass logsearch -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN'# keytool -genkeypair -alias logsearch -keyalg RSA -keysize 2048 -keypass logsearch -storepass logsearch -validity 3650 -keystore /etc/security/serverKeys/logsearch.trustStore.jks -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN' -rfc.- In the above example i used CN=erie1.example.com as my LogSearch service is running on the same host.- Now we will need to make sure that those files has proper read permissions and ownership set corr

OverviewR is currently supported through livy interpreter by running %livy.sparkrEnvironmentI did test the below solution withHDP 2.5.0.0-1245 and Ambari 2.4.0.1HDP 2.5.3.0-37 and Ambari 2.4.2.0Steps To Follow1. Install R on all YARN worker nodes$ yum install R-devel libcurl-devel openssl-devel2. Confirm R works fine[root@dkhdp251 zeppelin]# R -e "print(1+1)"R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch"Copyright (C) 2016 The R Foundation for Statistical ComputingPlatform: x86_64-redhat-linux-gnu (64-bit)R is free software and comes with ABSOLUTELY NO WARRANTY.You are welcome to redistribute it under certain conditions.Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English localeR is a collaborative project with many contributors.Type 'contributors()' for more information and'citation()' on how to cite R or R packages in publications.Type 'demo()' for some demos, 'help()' for on-line help, or'help.start()' for an HTML brow

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-wor

There is a two part process before any access to NiFi UI is possible:1. Authentication:By default NiFi will use a user/server's SSL certificate when provided in the connection to authenticate. When NO user/server certificate is presented, NiFi will then look for a Kerberos TGT (If Spnego has been configured in NiFi). Finally, if neither of the above where present in the connection, NiFi will use the login identity provider (if configured). Login identity providers include either ldap or kerberos. With both of these options, NiFi will present users with a login screen.2. Authorization:Authorization is the mechanism that controls what features and components authenticated users are granted access. The default authorizer NiFi will use is the internal file based authorizer. There is an option to configure NiFi to use Ranger as the authorizer instead.The intent of this article is not to discuss how to setup NiFi to use any of the Authentication or Authorizer options. This article covers

Centralized Cache Management in HDFS is a mechanism that explicitly caches specific files or directories in memory for improved performance. This is useful for relatively small files that are accessed repeatedly. For example, reference/lookup tables or fact tables that are used in many joins. Once enabled, HDFS will automatically cache selected files, and periodically check for changes and recache the files.While HDFS and the underlying file system do some caching of files when memory is available, explicit caching using Centralized Cache Management prevents the data from being evicted from memory when processes consume all of the physical memory. As a corollary of this, if you ARE working on a lightly loaded system where there is free memory, you may not see any performance improvement from this method, as the data was already in disk cache. So, your performance testing needs to stress the system.Let’s look at some key terms and concepts:Cache PoolsA cache pool is an administrative en

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 10: https://community.hortonworks.com/articles/85354/apache-ambari-workflow-manager-view-

In this article we will talk about one of the best feature of Ambari 2.4 in which we will be doing the ambari "setup-security" in non interactive mode to enable the HTTPs in ambari server. I assume that you have alredy created the ambari-server keys and certificate using open ssl. So we will see how can we do the HTTPs setup using a single line command:ambari-server setup-security --security-option=setup-https --api-ssl=true --api-ssl-port=8443 --import-cert-path=/etc/ambari-server/certs/sandbox.hortonworks.com.crt --import-key-path=/etc/ambari-server/certs/sandbox.hortonworks.com.key --pem-password=hadoopOutput of the above command can be as following, Notice that it will run in non interactive mode.Using python /usr/bin/pythonSecurity setup options...Do you want to configure HTTPS [y/n] (y)? SSL port [8443] ? Please enter password for Private Key: Importing and saving Certificate...done.Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.t

IntroductionGeospatial data is generated in huge volumes with the rise of the Internet of Things. IoT sensor networks are pushing the geospatial data rates even higher. There has been an explosion of sensor networks on the ground, mobile devices carried by people or mounted on vehicles, drones flying overhead, tethered aerostats (such as Google’s Project Loon), atmosats at high altitude, and microsats in orbit.OpportunityGeospatial analytics can provide us with the tools and methods we need to make sense of all that data and put it to use in solving problems we face at all scales.ChallengesGeospatial work requires atypical data types (e.g., points, shapefiles, map projections), potentially many layers of detail to process and visualize, and specialized algorithms—not your typical ETL (extract, transform, load) or reporting work. Apache Spark Role in Geospatial DevelopmentWhile Spark might seem to be influencing the evolution of accessory tools, it’s also becoming a default in the geosp