Community Articles

IntroductionApache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data. In this article we explore the support for windowing operations and stateful processing that were recently added to Apache Storm.Windowing support Windowing computations is one of the common use cases in stream processing where the unbounded stream of data is split into finite sets based on some criteria (e.g. time) and computation is applied on each group of events. An example would be to compute the top trending twitter topic in the last hour. Windowing is primarily used for aggregations, joins, pattern matching and more. Windows can be seen as an in-memory table where events are added and evicted based on some policies. So far Apache Storm relied on developers to built their own windowing logic. There were no recommended or high level abstractions that developers could use to define a Window in a standard way in a Topology

In previous article of the sereies, Enriching Telemetry Events, we walked through how to enrich a domain element of a given telemetry event with WhoIs data like home country, company associated with domain, etc. In this article, we will enrich with a special type of data called threat intel feeds. When a given telemetry event matches data in a threat Intel feed, an alert is generated. Again, the customers requirement are the following: The proxy events from Squid logs needs to ingested in real-time.The proxy logs has to be parsed into a standardized JSON structure that Metron can understand.In real-time, the squid proxy event needs to be enriched so that the domain named are enriched with the IP informationIn real-time, the IP with in the proxy event must be checked against for threat intel feeds.If there is a threat intel hit, an alert needs to be raised.The end user must be able to see the new telemetry events and the alerts from the new data source.All of this requirements will nee

I'm a long-time user of Apache Bigtop. My experience with Hadoop and Bigtop predates Ambari. I started using Bigtop with version 0.3. I remember pulling bigtop.repo file and install Hadoop, Pig and Hive for some quick development. Bigtop makes it convenient and easy. Bigtop has matured since then and there are now multiple ways of deployment. There's still a way to pull repo and install manually but there's better ways now with Vagrant and Docker. I won't rehash how to deploy Bigtop using Docker as it was beautifly described here. Admittedly, I'm running it on Mac and was not able to provision a cluster using Docker. I did not try with non-OSX. This post is about Vagrant. Let's get started:Install VirtualBox and VagrantDownload 1.1.0 releasewget http://www.apache.org/dist/bigtop/bigtop-1.1.0/bigtop-1.1.0-project.tar.gzuncompress the tarballtar -xvzf bigtop-1.1.0-project.tar.gzchange directory to bigtop-1.1.0/bigtop-deploy/vm/vagrant-puppet-vmcd bigtop-1.1.0/bigtop-deploy/vm/vagrant-pup

aka YAND; Yet Another Nifi Demo The recording explains it all really, but see below for detailed instructions on how to duplicate it. Recording of SAMS in action ## Background I recently attended YASK, the Hortonworker sales kickoff, and as a part of the training we were to spend some time learning and presenting the corporate sales pitch. This is 35 slides covering the major story points of the business, and takes about 20mins to blast through if you don't stop for questions. It isn't for field use, but it's a good educational tool to get new people up to speed. Instead of heading back to my hotel room the night before, I got distracted towards the pub by a regional SE director. After a couple of beers we got around to discussing the schedule for the next day, and I joked that it would be pretty funny to use Nifi to do the pitch a la @Jeremy Dyer. Grant (the director) uttered the immortal phrase "I'd be really disappointed if you didn't", and now we're here. ## Credits P

One of the key design principles of Apache Metron is that it should be easily extensible. We envision many users using Metron as a platform and building custom capabilities on top of it; one of which will be to add new telemetry data sources. In this multi-part article series, we will walk you through how to add a new data telemetry data source: Squid proxy logs.This multi-part article series consists of the following: This Article: Sets up the use case for this multi-part article seriesUse Case 1: Collecting and Parsing Telemetry Events - This tutorial walks you through how to collect/ingest events into Metron and then parse them.Use Case 2: Enriching Telemetry Data - Describes how to enrich elements of telemetry events with Apache Metron.Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds - Describes how to add new threat intel feeds to the system and how those feeds can be used to cross-reference every telemetry event that comes in. When a hit occurs, an alert will be ge

User demouser needs access to /landingzone. Super user HDFS created a directory called /landingzone and user demouser does not have access to it. We will use Ranger to control the access.

Data Federation discussions are becoming more and more common place as organizations embark on their Big Data Journey. New data platforms like the Hortonworks Connected platform (HDP+HDF), NoSQL, and NewSQL data stores are reducing the cost and difficulty of storing and working with vast volumes of data. This is empowering organizations to leverage and monetize their data like never before. However, legacy data infrastructures still play an important role in the overall technology architecture. In order to fully realize the power of the new and traditional data platforms, it is often necessary to integrate the data. One obvious approach is to simply move the data from where it sits in the existing data platform over to the target data platform. However, in many cases it is desirable to leave the data in place and enable a "Federation" tier to act as a single point of access to data from multiple sources. For details on the concepts and implementation of Data Federation see https://comm

Metron TP1 Features The following are key capabilities available in Metron TP1 broken up across its four key functional themes.How do I get Started?You can spin up the Metron TP1 in two ways: Ansible based Vagrant Single Node VM Install This the best place to play with Metron First.Detailed instructions how to do the install can be found in the following HCC Article: Apache Metron TP 1 Install Instructions- Single Node Vagrant Deployment Fully Automated 10 Node Ansible Based Install on AWS using Ambari Blueprints and AWS APIs If you want a more realistic setup of the Metron app, use this approach. Keep in mind that this install will spin up 10 m4.xlarge EC2 instance by defaultDetailed instructions how to do the install can be found in the following HCC Article: Apache Metron - First Steps in the CloudWhere do I get Help?Hortonworks has created new Track called CyberSecurity in the Hortonworks Community Connection (HCC). The link to the this new track in HCC is the following: HCC Cyb

Falcon by default coes with authorization turned off. To turn on set the following through ambari falcon config:*.falcon.security.authorization.enabled = true*.falcon.security.authorization.superusergroup = <linux group>In my example I am using linux group usersIn my example oozie,ambari-qa,tez,falcon,hue,guest are in the group "users". The purpose of this group is to only allow users within this group to view, edit, and delete each others material. Any user outside this group should not have access.Now logging in a falcon who is part of users group:Falcon user has created a cluster "authTest" and feed "feed1" Lets view it:Great so falcon is see the feed and cluster. Now lets go in with user hdfs who is NOT part of group usersLogged in as user hdfs who is NOT part of group users. This user will do a simple search for everything cluster/feed entity which exist in falcon.So hdfs user search does not return anything since the use is not allowed. Now lets log in with user tez wh

I have seen phantom or false alerts being generated by Ambari Alerts that clear on the next testing cycle without any evidence of an actual outage of the component. This issue may be related to a race condition within the python scripts where the check for whether the kerberos ticket is still valid occurs right at the end of the 5 minute lifetime and becomes invalid before the actual alert check can be run against the component. This causes the script to error-out which is taken as a failure of the alert check. In order to try to avoid this race condition, it has been suggested and there has been some early evidence to support the fact that increasing the kerberos lifespan setting to be a different, longer time than the alert check interval (which by default is 5 minutes).The parameter "-l 5m" (that's a lower-case L) is used within these two python scripts to set the lifetime to 5 minutes:/usr/lib/ambari-server/lib/resource_management/libraries/functions/curl_krb_request.py /var/lib

When performing a Rolling or Express Upgrade, failures can naturally happen because large clusters are bound to have problematic hosts. Here are 10 easy tips to prevent, diagnose and fix errors.Before upgrading the stack ... 1. Always upgrade Ambari to the most recent version, even if it's a dot release. Often, there are fixes and optimizations that make the stack upgrade smoother. 2. Ensure all services are up, service checks are passing, there are no critical alerts, etc. This helps ensure that the cluster is fully operational and helps to isolate any failures. 3. Pre-Install the bits and make sure all hosts have enough disk space. You can check that the version is found on all hosts. E.g., hdp-select versions | grep 2.5.0.0 | sort | tail -1 4. Do not ignore warnings. Starting in Ambari 2.2.2, there's a flag in ambari.properties file that allows users to bypass PreCheck errors, make sure it is either not present or set to false, stack.upgrade.bypass.prechecks=false 5. T

IntroductionApache Tez has become a very important framework and API to support batch and interactive over terabytes and petabytes of data for many engines within HDP such as Pig, Hive, Java. Cascading and others, with performance advantages at scale over Map Reduce and even Spark at certain volumes of data.For more on Apache Tez see http://hortonworks.com/hadoop/tez/This article is meant to outline the best practice in configuring and tuning Tez, and why you would set certain values in certain properties to get performance at scale, with step by step instructions.With this in place you would hopefully prevent out of memory errors when you execute your Hive Queries or Pig Scripts as seen in https://community.hortonworks.com/questions/5780/hive-on-tez-query-map-output-outofmemoryerror-java.html https://community.hortonworks.com/questions/12067/what-is-the-workaround-when-getting-hive-outofmemo.htmlTez Memory DemystifiedI find a diagram usually helps to understand why you would set cert

So you've just got all excited about the Metron Tech Preview, have downloaded the code and maybe have even followed the Apache Metron - First Steps in the Cloud article, but then EC2 instances pop up where you don't expect to see them!Unfortunately at this stage the Metron Tech Preview deployment ignores some of the systemwide settings that you might have in your .aws directory including the preferred region and availability zone.Thankfully all the settings are within the setup scripts instead so this is easy enough to correct.There are two settings I'd suggest updating:./deployment/amazon-ec2/conf/defaults.yamlContains the key settings you will need to change towards the top of the file:# ec2env: test-metronregion: us-west-2instance_type: m4.xlargeimage: ami-05cf2265For example, to change this to run in the EU Ireland region your config settings would need to look like this:# ec2env: mynamehere-metronregion: eu-west-1instance_type: m4.xlargeimage: ami-edb9069eNote that I've also ch

In a multiple application environment (MapReduce, Tez, others), it can appear that some application get stuck (blocked, deadlocked), it is likely your YARN / MAPREDUCE settings need to be reviewed. First of all, setup YARN queue, check the value of mapreduce.job.reduce.slowstart.completedmaps (interesting to change it to 0.9) and enable preemption.

IntroductionHBase replication provides with a way of replicating HBase data from one cluster to another by adding the remote Zookeeper quorum as remote peer.Configuration on the clusterFirst of all, it is necessary to set thehbase.replication property to true. Then add the remote peer through hbase shell. The peer id can be any short name:For example:add_peer '1', "hdpdstzk01.machine.domain, hdpdstzk02.machine.domain, hdpdstzk03.machine.domain:2181:/hbase-secure"(If using Kerberos then the right JAAS configuration needs to be used, or it would be required to have the hbase service keytab in the cache to authenticate correctly against Zookeeper through SASL).Configuration on the tablesReplication is set at table and column family level by setting the propertyREPLICATION_SCOPE to ‘1’. The default value that tables get created with if not specified is ‘0’, which means no replication. If applying on already existing tables, then they need to be disabled, then the property added through alt

After completing this tutorial you will understand how to: leverage Spark to infer a schema on a CSV dataset and persist it to Hive without explicitly declaring the DDL deploy the Spark Thrift Server on the Hortonworks Sandbox connect and ODBC tool (Tableau) to the Spark Thrift Server via the Hive ODBC driver, leveraging caching for ad-hoc visualization Assumption 1: It is assumed that you have downloaded and deployed the Hortonworks sandbox, installed the Hive ODBC driver on your host machine, and installed Tableau (or your preferred ODBC-based reporting tool). Assumption 2: Please ensure that your host machine's /etc/hosts file has the appropriate entry mapping sandbox.hortonworks.com to the IP of your sandbox (e.g., 172.16.35.171 sandbox.hortonworks.com sandbox).Deploying the Spark Thrift Server Within Ambari, click on the Hosts tab and then select the sandbox.hortonworks.com node from the list. Now you can click “Add” and choose Spark Thrift Server from the

Automated deployment of a fresh HDP cluster that includes Zeppelin (via blueprints)Background:The Zeppelin Ambari service has been updated and now supports installing latest Zeppelin version (0.5.5) on HDP using blueprints to automate creation of 'data science' cluster. SequenceIQ team have a datascientist blueprint that installs Zeppelin, but based on my conversations with @lpapp, its based on an older version of the Ambari service so does not install the latest or support as many options.Below is a writeup of how to deploy Zeppelin via blueprints. Note that if you already have a cluster running, you should just use the Add service wizard in Ambari to deploy Zeppelin using the steps on the githubPurpose: Sample steps below for installing a 4-node HDP cluster that includes Zeppelin, using Ambari blueprints and Ambari bootstrap scripts (by @Sean Roberts)Pre-reqs: Bring up 4 VMs imaged with RHEL/CentOS 6.5 or later (e.g. called node1-4 in this case). Note that the VMs should not already

First you need to get the ID of the alert definition in your system:curl -H "X-Requested-By: ambari" -u admin:<password> -X GET http://<ambari_host>:<ambari_port>/api/v1/clusters/<cluster_name>/alert_definitions?AlertDefinition/name=ambari_agent_disk_usageGet the alert definition with the id from prevoius step:curl -H "X-Requested-By: ambari" -u admin:<password> -X GET http://<ambari_host>:<ambari_port>/api/v1/clusters/<cluster_name>/alert_definitions/61Set the new thresholds by changing the JSON alert definition:curl -H "X-Requested-By: ambari" -u admin:<password> -i -X PUT -d '{ "AlertDefinition":{ "cluster_name":"xyz", "component_name":"AMBARI_AGENT", "description":"This host-level alert is triggered if the amount of disk space used on a host goes above specific thresholds. The default values are 85% for WARNING and 90% for CRITICAL.", "enabled":true, "id":61, "ignore_host":false, "interval":1

Lab OverviewIn this lab, we will learn to: Configure Solr to store indexes in HDFSCreate a solr cluster of 2 solr instances running on port 8983 and 8984Index documents in HDFS using the Hadoop connectorsUse Solr to search documentsPre-Requisite The lab is designed for the HDP Sandbox. Download the HDP Sandbox here, import into VMWare Fusion and start the VMLABStep 1 - Log into Sandbox After it boots up, find the IP address of the VM and add an entry into your machines hosts file e.g.192.168.191.241 sandbox.hortonworks.com sandbox Connect to the VM via SSH (root/hadoop), correct the /etc/hosts entryssh root@sandbox.hortonworks.com If running on an Ambari installed HDP 2.3 cluster (instead of sandbox), run the below to install HDPsearchyum install -y lucidworks-hdpsearchsudo -u hdfs hadoop fs -mkdir /user/solrsudo -u hdfs hadoop fs -chown solr /user/solr If running on HDP 2.3 sandbox, run belowchown -R solr:solr /opt/lucidworks-hdpsearch Run remaining steps as solrsu solrStep 2 - Co

See Slides & video for the session.

This was tested on Yosemite 10.10.5 1) Install NiFi on your MacOS: http://docs.hortonworks.com/HDPDocuments/HDF1/HDF-1.2/bk_HDF_InstallSetup/content/ch_HDF_installing.html 2) Setup your machine to foward syslog messages to port 1514. Backup your current syslog configuration. mv /etc/syslog.conf /etc/syslog.conf.bkp Edit your syslog.conf file to send all messages to UDP localhost port 1514 sudo vi /etc/syslog.conf Add the following entry to /etc/syslog.conf *.* @127.0.0.1:1514Restart syslogd sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist Confirm syslogd is running. A result should display a process id (PID) for /usr/sbin/syslogd ps -ef | grep syslogd 3) Test with NiFi. Add a ListenSyslog processor to the canvas with following settings: Protocol: UDP Port: 1514 Local Network Interface: lo0 Connect the ListenSyslog process to an output port and have the r

Setting up Hortonworks Dataflow (HDF) to work with kerberized Kafka in Hortonworks Data Platform (HDP)HDF 1.2 does not contain the same Kafka client libraries as the Apache NiFi version. HDF Kafka libraries are specifically designed to work with the Kafka versions supplied with HDP.The following Kafka support matrix breaks down what is supported in each Kafka version:*** (Apache) refers to the Kafka version downloadable from the Apache website.For newer versions of HDF (1.1.2+), NiFi uses zookeeper to maintain cluster wide state. So the following only applies if this is a HDF NiFi cluster:1. If a NiFi cluster has been setup to use a kerberized external or internal zookeeper for state, every kerberized connection to any other zookeeper would require using the same keytab and principal. For example a kerberized embedded zookeeper in NiFi would need to be configured to use the same client keytab and principal you want to use to authenticate with a say a Kafka zookeeper. 2. If

This is an extension of Starting Spark jobs directly via YARN REST APIAssume the cluster is kerberized and the only access is Knox. Further assume that Knox uses Basic Authentication and we have user and password of the user to start the Spark job.The overall idea is to callcurl with <<user>>:<<password>> as Basic Authentication ==> Knox (verifying user:password against LDAP or AD) ==> Resource Manager (YARN REST API) with kerberos principal1) Create and distribute a keytab The user that should run the spark job also needs to have a kerberos principal. For this principle create a keytab on one machine:[root@KDC-HOST ~]$ kadminkadmin> xst -k /etc/security/keytabs/<<primaryName>>.keytab <<primaryName>>@<<REALM>>Use your REALM and an appropriate primaryName of your principleThen distribute this keytab to all other machines in the cluster, copy to /etc/security/keytabs and set permissions[root@CLUSTER-HOST ~]$ chown <&

Snakebite is a pure python based HDFS client which is much faster than the classic hdfs client operations(hadoop fs / hdfs dfs).Snakebite is developed by Spotify. More details can be found at https://github.com/spotify/snakebiteI am using HDP 2.4 and Ambari 2.2.1.1 on CentOS/1). Install Python Pip. yum install python-pip2).Install Snakebite.pip install snakebiteNow snakebite is installed. You can run below sample commands.snakebite ls /tmpsnakebite ls /Use snakebite help for more options.Happy Snakebiting!

Apache Drill is an open source, low-latency query engine for Hadoop that delivers secure, interactive SQL analytics at petabyte scale. With the ability to discover schemas on-the-fly, Drill is a pioneer in delivering self-service data exploration capabilities on data stored in multiple formats in files or NoSQL databases. By adhering to ANSI SQL standards, Drill does not require a learning curve and integrates seamlessly with visualization tools. SourceTutorial:Grab latest version of drillwget http://getdrill.org/drill/download/apache-drill-1.2.0.tar.gz tar xvfz apache-drill-1.2.0.tar.gz /root/drill/apache-drill-1.2.0[root@node1 apache-drill-1.2.0]# cd bin/Start Drill in distributed mode ( You can start in embedded mode too)[root@node1 conf]# ../bin/drillbit.shstartstarting drillbit, logging to /root/drill/apache-drill-1.2.0/log/drillbit.outDrill Web Console - http://host:8047Enable storage pluginsClick Storage --> Enable hbase,hive,mongoModify storage plugin for Hive and HBase as