Resources

Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.htmlPart 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.htmlPart 3: https://community.hortonworks.com/articles/82988/apache-ambari-workflow-designer-view-for-apache-oo-1.htmlPart 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-f

Security is very important at all, but most important in production environment. Cloudbreak delivers default settings, but those settings are purposing easy first dating, not security. So first of all what you can do is block all communication ports except 443 on the firewall / security group / what ever you use. Hardly recommended to open SSH port (usually 22) if you have to log in to the Cloudbreak host remotely. Second step is to configure Profile before starting Cloudbreak for the first time. First execute the following command in the directory where you want to store Cloudbreak related files: echo export PUBLIC_IP=[the ip or hostname to bind] > Profile After you have a base Profile you should add custom properties to the Profile file: export UAA_DEFAULT_SECRET='[custom secret]'export UAA_DEFAULT_USER_EMAIL='[default admin email address]'export UAA_DEFAULT_USER_PW='[default admin password]'export UAA_DEFAULT_USER_FIRSTNAME='[default admin first name]'export UAA_DEFAULT_USER_

1. Host a Web Page (index.html) via HTTP GET with 200 OK Status 2. Receive POST from that page via AJAX with browser data 3. Extract Content and Attributes 4. Build a JSON file of HTTP data 5. Store it To accept location in a phone or modern browser you must be running SSL. So I added that for this HTTP Request. Use openssl to create your 2048 RSA X509, PKCS12, JKS Keystore, Import Trust Store and import in browser Your web page can be any web page, just POST back via AJAX or Form Submit. <html><head><title>NiFi Browser Data Acquisition</title><body><script>// Usagewindow.onload = function() { navigator.getBattery().then(function(battery) { console.log(battery.level); battery.addEventListener('levelchange', function() { console.log(this.level); });});};////////////// print these var latitude = ""; var longitude = ""; var ips = ""; var batteryInfo = ""; var screenInfo = screen.width +","+ screen.height + "," + scr

The hadoop.security.auth_to_local under Advanced core-site.xml in HDFS service can be modified to remove spaces in user names. This translation rule is a modification of the default ruleThe default rule has a line which removes the domain name RULE:[1:$1@$0](.*@MYDOMAIN.COM)s/@.*//In order to translate the user name with Spaces "John Doe@MYDOMAIN.COM" to "John_Doe" The following rule can be usedRULE:[1:$1](.* .*)s/ /_/g/L RULE:[1:$1@$0](.*@MYDOMAIN.COM)s/@.*//The first rule replaces the Space with the Underscore while retaining the MYDOMAIN.COM and allows second rule also to be respected. This results in successful replacement of usernames with spaces to underscores.

Ambari Infra Solr Fails to start with "Unrecognized option: --secure-solr-znode" check the versions of ambari-infra like below, notice the version mis-match between client and server.[root@master1 zookeeper]# rpm -qa | grep ambariambari-infra-solr-client-2.4.2.0-136.x86_64ambari-infra-solr-2.5.0.0-771.x86_64Upgrade ambari-infra-solr-client

Download the HDC cli from the cloud controller Although it's possible to download the cli from the cloud controller UI we're going to use CURL for this, since we're about to automate the process. All we need is the address of the cloud controller (we make sure with $(uname) that we download the appropriate binary for our OS, note: all of the instance references are examples, there is no real running aws instance): curl -kL https://ec2-34-251-140-175.eu-west-1.compute.amazonaws.com/hdc-cli_$(uname)_x86_64.tgz | tar -xzOnce it's downloaded we can verify it's version: ./hdc --versionhdc version 1.13.0-2017-02-09T08:56:06 Configure the cli to use our cloud controllerThe cli can connect to multiple cloud controllers by providing it's address and credentials. Each command has parameters to provide these informations (even can be configured with environment variables): --server value server address [$CB_SERVER_ADDRESS] --username value user name (e-mail address) [$CB_USER_NAME] --pas

Q: How and when was Apache Ranger incubated and what is its current status in Apache? Is it a top level project (TLP)?About two years ago, Hortonworks donated the entire code base of about 440,000 lines from its XA Secure acquisition to the Apache Software Foundation (ASF) in order to help jump start Apache Ranger as an Apache Incubator project. It’s been a rewarding journey, from the first version of Apache Ranger released in November 2014, to achieving a major milestone by graduating to a top level Apache open source project. We are excited to report that recently ASF graduated Apache Ranger to a Top-Level Project (TLP) “signifying that the project's community and products have been well-governed under the ASF's meritocratic process and principles.” We, at Hortonworks, extend our hearty congratulations to the Apache Ranger community, without whom TLP status couldn’t have been achieved. This was a great community effort, representing the best of open source and the “Apache Way”!Q: Why

When synchronizing users from AD or LDAP to Ranger using usersync, It is possible that there are spaces in usernames. These spaces in user names and Group Names may be replaced by SSSD to use underscores. (Example SSSD config section below. [sssd]override_space = _ Ranger needs to synchronize users similarly to avoid issues in access permission setup. To enable Ranger to perform this conversion , the following flags can be added to "Ranger --> Configs --> Advanced --> custom ranger-ug-site" By adding 2 propertiesranger.usersync.mapping.groupname.regex, ranger.usersync.mapping.username.regex These two properties can be set to s/ /_/gAfter setting these properties restart ranger service using Ambari and then restart Ranger user sync. User sync will add new user names with Underscores in the user name. Older usernames and group names with Spaces will need manual cleanup. Note: This f

MiNiFi C++ implemented the secure site to site raw socket transport using OpenSSL/TLS.PR MINIFI-184: Add Security Support https://github.com/apache/nifi-minifi-cpp/commit/63dbb8241e851068bff54ab8cef8310cc4a22cb5 implemented the sameIt is using both client and server certificate to do mutual authentication between client and server via OpenSSL/TLS.Please look at http://bryanbende.com/development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site about setting up the NiFi server site security configuration. Before you start to configure MiNiFi C++, you need to have client certificate PEM file, client private key PEM file, client CA certificate PEM file, passphrase to the client certificate. Client certificate PEM file, client private key PEM file can be combined into a single PEM file.You can use openssl tool to convert different certificates formats between PEM/PKCS, etc.Modify conf/minifi.properties to add Site to Site secure setting### Site2Site Security Configuration in minifi.prop

ProblemI have my shiro configured as[users] # List of users with their password allowed to access Zeppelin. # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections #admin = password1 admin = change #user1 = password2, role1, role2 #user2 = password3, role3 #user3 = password4, role2 # Sample LDAP configuration, for user Authentication, currently tested for single Realm [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = ou=my,ou=company,o=com ldapRealm.userDnTemplate = uid={0},ou=my,ou=company,o=com ldapRealm.contextFactory.url = ldap://<ldap_host>:389 ldapRealm.contextFactory.authenticationMechanism = SIMPLE # doc horton ldapRealm.contextFactory.systemUsername = <system_username_bind>ldapRealm.contextFactory.systemPassword = <system_username_password> #sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSess

Sometime we get a situation where we have to get lists of all long running and based on threshold we need to kill them.Also sometime we need to do it for a specific yarn queue. In such situation following script will help you to do your job.#!/bin/bashif [ "$#" -lt 1 ]; then echo "Usage: $0 <max_life_in_mins>" exit 1fiyarn application -list 2>/dev/null | grep "report" | grep "RUNNING" | awk '{print $1}' > job_list.txtfor jobId in `cat job_list.txt`dofinish_time=`yarn application -status $jobId 2>/dev/null | grep "Finish-Time" | awk '{print $NF}'`if [ $finish_time -ne 0 ]; then echo "App $jobId is not running" exit 1fitime_diff=`date +%s`-`yarn application -status $jobId 2>/dev/null | grep "Start-Time" | awk '{print $NF}' | sed 's!$!/1000!'`time_diff_in_mins=`echo "("$time_diff")/60" | bc`echo "App $jobId is running for $time_diff_in_mins min(s)"if [ $time_diff_in_mins -gt $1 ]; then echo "Killing app $jobId" yarn application -kill $jobIdelse echo "App $job

In the big data and distributed system world you may have done a world class job in dev and Unit testing, chances are you still did it on sample data moreover in dev you might not have done it in a truly distributed system but more on one machine simulating distribution. Hence on the prod cluster you could rely on logs and sometime you would also like to connect a remote debugger and other tools you are used to. In this blog post I will go over an example in Java and Eclipse. Partly because I use those and also because I see mostly scala examples so let’s give java a little love.Setting upI have installed a Hortonworks sandbox for this exercise, you can donwload it here. You will also need to open up a port to bind on, on the last post I used port 7777 i’ll stick with this here as well.First we will use a very standard wordcount example you can get in all the spark tutorials. Set up a quick maven project to bundle your jar and dependent libs you might be using. Notice the breakpoint on

Caveat: This feature has been validated by HWX engineer manually but we don't officially support it at the moment.Environment: HDP-2.5.3.0-37Ambari-2.4.2.0-136JDK1.8Kerberos enabledRanger enabledDue to security limitations, we can only launch flume agent processes in Ambari.STEP 1: Create/modify flume configuration file.Ambari -> Flume -> Configs -> flume.conf# Flume agent config#### Global ####demo.sources = logtcp logudpdemo.channels = kafka_channeldemo.sinks = sink#### Sources ####demo.sources.logtcp.type = multiport_syslogtcpdemo.sources.logtcp.ports = 9515demo.sources.logtcp.host = 0.0.0.0demo.sources.logtcp.keepFields = truedemo.sources.logtcp.selector.type=replicatingdemo.sources.logtcp.channels= kafka_channeldemo.sources.logudp.type = syslogudpdemo.sources.logudp.port = 9515demo.sources.logudp.host = 0.0.0.0demo.sources.logudp.keepFields = truedemo.sources.logudp.selector.type=replicatingdemo.sources.logudp.channels = kafka_channel#### Sinks ####demo.sinks.sink.type =

Problem When running$ ambari-server restartI can see "tones" of these messages in ambari-server.log27 Feb 2017 11:38:00,942 WARN [main] AbstractLifeCycle:204 - FAILED SelectChannelConnector@0.0.0.0:8080: java.net.BindException: Address already in usejava.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:463) at sun.nio.ch.Net.bind(Net.java:455) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.j

Objective This tutorial is designed to walk you through the process of creating a MiniFi flow to read data from a Sense HAT sensor on a Raspberry Pi 3. The MiniFi flow will push data to a remote NiFi instance running on your computer. The NiFi instance will push the data to Solr. While there are other tutorials and examples of using NiFi/MiniFi with a Raspberry Pi, most of those tutorials tend to use a more complicated sensor implementation. The Sense HAT is very easy to install and use.Prerequisites You should have a Raspberry Pi 3 Model B: Raspberry Pi 3 Model BI recommend a 16+GB SD card for your Raspberry Pi 3.Don't forget to expand the filesystem after the OS is installed: raspi-config You should have a Sense HAT: Sense HATYou should already have installed the Sense HAT on your Raspberry Pi 3. You should already have installed Raspbian Jessie Lite on your Raspberry Pi 3 SD card: Raspbian Jessie LiteThe instructions for installing a Raspberry Pi OS can be found here:

This also includes an analysis on the fly for showing odds on a Craps game. This example shows a simple use of Nifi - HDF - handling multiple streams of Dice data - each one simulating a separate Craps table – showing a Monte-Carlo simulation and results of a 1000 run – emulating throws each second.To demonstrate this capability we generate some random dice data, each stream generation uses independent thread. We throttle the threads to sleep for a second between throws, mainly to demonstrate an ongoing stream of data over time.Source for data generation:https://github.com/eorgad/Dice-nifi-streams-example/tree/master/Dice-nifi-stream-example/Dice-nifi-streams/srcWe use Nifi to create a streaming flow of that data as it is being generated.This simulation will use the following Nifi processors:HandleHttpRequest (Starts an HTTP Server and listens for HTTP Requests)RouteOnAttribute (Routes FlowFiles based on their Attributes using the Attribute Expression Language)ExecuteStreamCommand (Exe

Sometime you may have a requirement where you need last accessed time of file in hdfs, then you may get it via following ways:Option 1: Ranger Audit: You can get it via ranger audit but I would not prefer it because output of audit file would be confusing due to many tmp files and dirs which present in audit db.So now to solve this requirement and to fulfill my purpose I have used java APIs and done my work very efficient and in a vary cleaned way. Option 2: Use java and build a small program. With the help of java program you can get some other useful information about hdfs file also in few number of lines. Step 1 : Create java program :package com.saurabh;import java.io.*;import java.util.*;import java.net.*;import java.nio.file.Files;import java.nio.file.Paths;import org.apache.hadoop.fs.*;import org.apache.hadoop.conf.*;import org.apache.hadoop.io.*;import org.apache.hadoop.mapred.*;import org.apache.hadoop.util.*;// For Date Conversion from long to human readable.import java.text.

HDF/NiFi is a key element in the architectural transformations with the future of data. It is a great tool to manage data flows but it also provides means to access data in motion as a stream source. This feature gives an easy way to expose data endpoints in data flows allowing users to access the data as soon as possible and perform desired analytics, online machine learning, etc. Use case description In this article, I will describe how to easily implement the following scenario: Use Apache NiFi to gather tweets filtered on specifics terms, Use Apache Spark streaming to retrieve the data, transform it and store it into an Apache Hive table, Use Apache Zeppelin to access the data and display real-time analytics. This example will use the US Presidential Election as a subject and will show: The frequency of tweets mentioning twitter accounts of one of the main candidates: Ted Cruz, Donald Trump, Hillary Clinton and Bernie Sanders. A sentiment analysis of the tweets with the pe

Steps to enable Trace logging in Ambari server for Oracle Database.Step1:1. Download ojdbc6_g.jar from Oracle JDBC driver trace jar and copy to /usr/share/java and make sure to have correct permissions. Note: take out the actual ojdbc.jar from the location.2.Create /etc/ambari-server/conf/oracle.properties.level=ALLoracle.jdbc.level=INFOoracle.jdbc.handlers=java.util.logging.ConsoleHandlerjava.util.logging.ConsoleHandler.level=INFOjava.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter3. Add "-Doracle.jdbc.Trace=true -Djava.util.logging.config.file=/etc/ambari-server/conf/oracle.properties" property in /var/lib/ambari-server/ambari-env.shexport AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djavto export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Doracle.jdbc.Trace=true -Djava.util.logging.config.file=/etc/ambari-server/con

Please find the attached "Storm/Kafka Best Practices Guide". This document is intended to be an unofficial guide to developing and deploying streaming applications using Storm and Kafka. This is not the official Hortonworks documentation, but meant to be a collection of best practices from teams implementing Storm and Kafka in Production. Pre-ReadingIf you are new to Storm Development or operations, please review the Tutorial under the Storm Documentation “Read these first” section. This document covers the basics of Storm components and operations.DevelopmentThe Storm developer has much opportunity to make the topology fast, efficient, and easier to run in Production by his Operations team. This section will discuss the best practices from the developer’s perspective to achieve these goals.Developing for highest PerformanceKeep the Tuple processing code as light as possible. Code so that the execute() method in your Bolts can run as quickly as possible. This will have multiplier effec

Introduction SQL development tools like DbVisualizer, SQuirreL SQL and DataGrip are popular options for database development. Although these tools don't offer native Hive support they can be easily configured to connect to Hive using JDBC. While connecting these tools to clusters without kerberos is relatively straightforward, the process of connecting them to kerberized clusters can be complex and error prone. This article, in combination with a project I created ( Hive JDBC Uber Jar), aim to simplify and standardize this process.Prerequisites There are a few key things that must be properly configured before attempting to connect to a kerberized cluster. A full description of these tasks is out of scope for this article, but at a high level, make sure that: You have downloaded the latest release of my Hive JDBC Uber Jar and placed it somewhere sensible DbVisualizer and/or DataGrip have been successfully installed on your workstation The krb5.conf file on your workstatio

One of the earliest improvement requests for the Apache NiFi project was to allow a user the ability to easily align components on the canvas. Well, in the upcoming 1.2 release it's HERE! In this article I will demonstrate how to align multiple selected components either vertically or horizontally.Previously, when building a flow a user would drop components on the canvas into a layout. Typically users will attempt to organize their data flow into rows and columns. This was a tedious task since aligning these components was a manual effort of dragging and dropping components to line them up 'just right'. Let's look at a crude example of such a layout:Let's see how we can organize this flow with the new 'click to align' features. First, let's align the rows:Ok great! Now let's align the columns:BOOM! And there you have it! A nicely aligned layout for an organized data flow. I hope you have found this tutorial useful and I look forward to your feedback.

NiFi works with FlowFiles. Every FlowFile that exists consists of two parts, FlowFile content and FlowFile Attributes. While the FlowFile's content lives on disk in the content repository, NiFi holds the "majority" of the FlowFile attribute data in the configured JVM heap memory space. I say "majority" because NiFi does swapping of Attributes to disk on any queue that contains over 20,000 FlowFiles (default, but can be changed in the nifi.properties).Once your NiFi is reporting OutOfMemory (OOM) Errors, there is no corrective action other then restarting NiFi. If changes are not made to your NiFi or dataflow, you are surely going to encounter this issue again and again.The default configuration for JVM heap in NiFi is only 512 MB. This value is set in the nifi-bootstrap.conf file.# JVM memory settingsjava.arg.2=-Xms512mjava.arg.3=-Xmx512mWhile the default may work for some dataflow, they are going to be undersized for others. Simply increasing these values till you stop seeing (OOM

Authentification 1- Kerberos : Kerberos is mandatory for prod environments, you can either use your AD embeded kerberos or install a new dedicated KDC - Kerberos must be in HARisk not doing the above : User impersonation for the services accounts ( jobs can be exported to run as super user permission )2 - Use a firewall to block all inbound traffic to the cluster- all sources / all ports except from the edge node ( Gateway )Risk not doing the above : Passwords in the wrong hands will systematically give access to the cluster3- Check the permissions of the keytabs, detailed this article here : Script to fix permissions and ownership of hadoop keytabs Risk not doing the above: utilisation by other cluster users 4- Use Knox for all API calls to the cluster.Benefits : inbound from "trusted" known machines, that requires authentification against an existing LDAP.Network 1 - The cluster must be in an isolated subnet - no interference with other networks, for security and thouroughputR

In order to setup HTTPs access for LogSearch Portal UI first of all we will need to create the keystore. We will also create truststore as well using the "$JAVA_HOME/bin/keytool" utility. In this example we will be using the self signed certificates.# mkdir -p /etc/security/serverKeys/# keytool -genkey -v -alias logsearch -keyalg RSA -keysize 1024 -keystore /etc/security/serverKeys/logsearch.keyStore.jks -validity 3650 -keypass logsearch -storepass logsearch -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN'# keytool -genkeypair -alias logsearch -keyalg RSA -keysize 2048 -keypass logsearch -storepass logsearch -validity 3650 -keystore /etc/security/serverKeys/logsearch.trustStore.jks -dname 'CN=erie1.example.com, OU=hwx, O=test, L=Pune, S=Maharashtra, C=IN' -rfc.- In the above example i used CN=erie1.example.com as my LogSearch service is running on the same host.- Now we will need to make sure that those files has proper read permissions and ownership set corr