Community Articles

Apache Spark has ignited an explosion of data exploration on very large data sets. Spark played a big role in making general purpose distributed compute accessible. Anyone with some level of skill in Python, Scala, Java, and now R, can just sit down and start exploring data at scale. It also democratized Data Science by offering ML as a series of black boxes. Training of Artificial Intelligence is now possible for those of us who do not have PHDs in Statistics and Mathematics. Now Spark SQL is also helping to bring data exploration to the business unit directly. In partnership with Apache Hive, Spark has been enabling users to explore very large data sets using SQL expression. However, in order to truly make Spark SQL available for ad-hoc access by business analysts using BI tools, fine grain security and governance are necessary. Spark provides strong authentication via Kerberos and wire encryption via SSL. However, up to this point, Authorization was only possible via HDFS ACLs. This

On RHEL/CentOS you might encounter an exception when trying to stop or restart Oozie :resource_management.core.exceptions.Fail: Execution of 'cd /var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-stop.sh' returned 1. -bash: line 0: cd: /var/tmp/oozie: No such file or directoryThis is likely because of a shell crontab /etc/cron.daily/tmpwatch which delete files/directories unmodified for 30d+[root@local ~]# cat /etc/cron.daily/tmpwatch#! /bin/shflags=-umc/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \ -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \ -X '/tmp/hsperfdata_*' 10d /tmp/usr/sbin/tmpwatch "$flags" 30d /var/tmpfor d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do if [ -d "$d" ]; then /usr/sbin/tmpwatch "$flags" -f 30d "$d" fidoneJust recreate the directory and you're good to go[root@local ~]# mkdir /var/tmp/oozie[root@local ~]# chown oozie:hadoop /var/tmp/oozie[root@local ~]# chmod 755 /var/tmp/ooz

Story: From the documentation I was able to add a service to existing stack definition in Ambari. Issue: But I was either not able to stop the service or delete the service just in case.https://cwiki.apache.org/confluence/display/AMBARI/Defining+a+Custom+Stack+and+ServicesHow did I solve the problem? 1.Create and Add Stack: cd /var/lib/ambari-server/resources/stacks/HDP/2.4/services 2.Create a directory that contains the service definition for SAMPLESRVmkdir /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRVcd /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV 3.Create a metainfo.xml as show in the link above. 4.With this we have a service name as SAMPLESRV and it contains SAMPLESRV_MASTER, SAMPLESRV_SLAVE and SAMPLESRV_CLIENT 5.Next we need to create the command scriptsmkdir –p /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scriptscd /var/lib/ambari-server/resources/stacks/HDP/2.4/services/SAMPLESRV/package/scripts6.Browse t

If you have ever tried to spawn multiple cloudbreak shells you may have run into an error. That is because the default "cbd util cloudbreak-shell" uses docker containers. The fastest work around of this is to use the Jars directly. These Jars can be remotely run from your personal machine or run on the cloudbreak machine itself.Prepping the cloudbreak machine(only needed if running jars locally on the AWS image)Log into your cloudbreak instance and go to /etc/yum.repos.dRemove the Centos-Base.repo file (this is a redhat machine and this can cause conflicts)Install java-8 (yum install java-1.8.0*)Change directory back to /home/cloudbreakDownloading the JarSet a global variable equal to your cloudbreak version (export CB_SHELL_VERSION=1.6.1)Download the jar (curl -o cloudbreak-shell.jar https://s3-eu-west-1.amazonaws.com/maven.sequenceiq.com/releases/com/sequenceiq/cloudbreak-shell/$CB_SHELL_VERSION/cloudbreak-shell-$CB_SHELL_VERSION.jar)Using the JarInteractive mode (java -jar ./cl

When using the Oozie Proxy job submission API for submitting the Oozie Hive, Sqoop and Pig actions. To pass any configuration to the action is required to be in below format.For Hive action:oozie.hive.options.size : The number of options you'll be passing to Pigoozie.hive.options.n : An argument to pass to Hive, the 'n' should be an integer starting with 0 to indicate the option number<property> <name>oozie.hive.options.1</name> <value>-Doozie.launcher.mapreduce.job.queuename=hive</value></property><property> <name>oozie.hive.options.0</name> <value>-Dmapreduce.job.queuename=hive</value></property><property> <name>oozie.hive.options.size</name> <value>2</value></property>For Pig Action: oozie.pig.options.size : The number of options you'll be passing to Pigoozie.pig.options.n : An argument to pass to Pig, the 'n' should be an integer starting with 0 to indicate

Today we will show how to interact with a NiFi instance to modify a flow at runtime via API.Pre-requisites NiFi installed and running on a localhost: https://nifi.apache.org/download.htmlGroovy - because its JSON builders and REST DSLs are great. If you are on a Mac, the easiest is to run brew install groovy. To install Homebrew (the superb Mac package manager), visit http://brew.sh/Full script is available on GitHub: https://github.com/aperepel/nifi-rest-api-tutorialNiFi REST API Docs: https://nifi.apache.org/docs/nifi-docs/rest-api/index.htmlHere’s the test flow we will be working with today:Prepare the test flow: Add a PutFile component to the canvas.Rename the processor to Save File (right-click -> Configure -> Settings -> Name field). We will be using this name to look up the processor later via API.Add a GetHTTP processor and create a connection from GetHTTP to ‘Save File’. GetHTTP settings can be ignored for now, the Save File processor simply needs an input connected.S

Raspberry PIs and other small devices often have cameras or can have camera's attached. Raspberry Pi's have cheap camera add-ons that can ingest still images and videos (https://www.raspberrypi.org/products/camera-module/). Using a simple Python script we can ingest images and then ingest them into our central Hadoop Data Lake. This is a nice simple use case for Connected Data Platforms with both Data in Motion and Data at Rest. This data can be processed in-line with Deep Learning Libraries like TensorFlow for image recognition and assessment. Using OpenCV and other tools we can process in-motion and look for issues like security breaches, leaks and other events. The most difficult part is the Python code which reads from camera, adds a watermark, converts to bytes, sends to MQTT and then ftps to an FTP server. I do both since networking is always tricky. You could also add if it fails to connect to either, store to a directory on a mapped USB drive. Once network returns

GoalThe goal of of this article is to get familiar with the most convenient method of accessing HAWQ Database i.e. using psql native clientAssumptions HAWQ Database is available and activePSQL Native Client (Available for download @ https://network.pivotal.io)PSQL or "psql" (the binary command) is the ideal choice to connect to HAWQ database/warehouse because HAWQ itself is based on Postgres database. Steps to configure psql on Client/Edge node: Make a note of the Public IP address for the Client/Edge Node, this will be needed later. Access to HAWQ is available by default from HAWQ master host, make a note of HAWQ database name and user you would want to use for enabling database access. To identify name of database (psql -l) from the gpadmin user prompt in linuxTo identify available user names "psql -c \du" Download the HAWQ installation binaries from the following site https://network.pivotal.io/products/pivotal-hdb#/releases/3466/file_groups/520Follow this guide to setup the HAWQ r

Thanks to deep collaboration between Hortonworks and Isilon engineering teams, we have come up with the below table that shows the status of Ambari ViewsThrough Ambari 2.2.2 Ambari Version Kerberos Enabled WebHDFS* Enabled Ambari View Works? Remarks Through Ambari 2.2.2 Y Y No Does not work with Kerberos enabled because OneFS implementation of WebHDFS does not support chunked encoding. However we believe that if we use HTTPFS in front , it should work. This has not been officially tested though. Through Ambari 2.2.2 Y N No Does not work in this combination . This is default behavior of Ambari views which does not support Kerberos without WebHDFS. Through Ambari 2.2.2 N Y Yes If the default fs.defaultfs is set to "hdfs" in core-site.xml, then it works. The default value in Ambari is: hdfs:// <rpc address, so it works by default. if you need to use custom Hive View Cluster

livy documentation states to add jars with the interpreter key value livy.spark.jars. This is applicable when running livy interpreter in yarn-cluster mode only (livy.spark.master). This can be done in local or yarn-client mode as well however by adding the jars to the /usr/hdp/<version>/livy/repl-jars directory. This will add the jars to the spark submit command run by livy when executing within a notebook in modes other than yarn-cluster.

Some people say I must have a bot to read and reply to email at all crazy hours of the day. An awesome email assistant, well I decided to prototype it. This is the first piece. After this I will add some Spark machine learning to intelligently reply to emails from a list of pretrained responses. With supervised learning it will learn what emails to send to who, based on Subject, From, Body Content, attachments, time of day, sender domain and many other variables. For now, it just reads some emails and checks for a hard coded subject. I could use this to trigger other processes, such as running a batch Spark job. Since most people send and use HTML email (that's what Outlook, Outlook.com, Gmail do), I will send and receive HTML emails as to make it look more legit. I could also run my fortune script and return that as my email content. Making me sound wise, or pull in a random selection of tweets about Hadoop or even recent news. Making the email very current and fresh. Snippet Example

Protect Your Cloud Big Data AssetsStep 1: Do not put anything into the cloud unless you have a CISO, Chieft Security Architect, Certified Cloud Administrator, full understanding of your PII and private data, a Lawyer to defend you against the coming lawsuits, full understanding of Hadoop, Hadoop Certified Administrators, a Hadoop premier support contract, a security plan, full understanding of your Hadoop architecture and layout. Step 2: Study all running services in Ambari. Step 3: Confirm and check all of your TCP/IP ports. Hadoop has a lot of them!Step 4: if you are not using a service, do not run it.Step 5: By default, disable all access to everything, always. Only open ports and access when something and someone critical cannot access them. Step 6: SSL, SSH, VPN and Encryption Everywhere.Step 7: Run Knox! Set it up correctly.Step 8: Run Kali and audit all your IPs and ports.Step 9: Use Kali hacking tools to attempt to access all your web ports, shells and other

Protect Your Cloud Big Data AssetsStep 1: Do not put anything into the cloud unless you have a CISO, Chieft Security Architect, Certified Cloud Administrator, full understanding of your PII and private data, a Lawyer to defend you against the coming lawsuits, full understanding of Hadoop, Hadoop Certified Administrators, a Hadoop premier support contract, a security plan, full understanding of your Hadoop architecture and layout. Step 2: Study all running services in Ambari. Step 3: Confirm and check all of your TCP/IP ports. Hadoop has a lot of them!Step 4: if you are not using a service, do not run it.Step 5: By default, disable all access to everything, always. Only open ports and access when something and someone critical cannot access them. Step 6: SSL, SSH, VPN and Encryption Everywhere.Step 7: Run Knox! Set it up correctly.Step 8: Run Kali and audit all your IPs and ports.Step 9: Use Kali hacking tools to attempt to access all your web ports, shells and othe

Objective This tutorial is intended to walk you through the process of troubleshooting why events are missing from the Metron Kibana dashboard when using the quick-dev-platform. Because of the constrained resources and added complexity of the virtualized environment, sometimes the vagrant instance doesn't come up in a healthy state. Prerequisites You should have already cloned the Metron git repo. Metron You should have already deployed the Metron quick-dev-platform. quick-dev-platform The expectation is you have already started Metron once and shut it down via a vagrant halt. Now you are trying to start Metron again via vagrant up, but things don't seem to be working properly. These troubleshooting steps are also be helpful during the initial start of the VM via the run.sh script. Scope Theses troubleshooting steps were performed in the following environment: Mac OS X 10.11.6 Vagrant 1.8.6 VirtualBox 5.1.6 Python 2.7.12 (Anaconda distribution) Steps Start Metron quick-dev-platform

The Apache Phoenix QueryServer (PQS) is a relatively new (early 2015) component that has been added. Previously, users of Phoenix had at least two things in common: the client was written in Java and the Phoenix client libraries were included in that client. As Phoenix continues to grow as a dominant player in the "Hadoop OLTP" space, the need to eliminate these two burdens became apparent. PQS was the solution chosen to address these problems.PQS specifically refers to an HTTP server which can interact with a custom JDBC driver that translates JDBC API calls to HTTP requests. These components are provided by an upstream Apache project, Apache Calcite, specifically the Calcite component referred to as Avatica. Originally, the HTTP requests were JSON; however recent work (mid-2015) has also shown that serialization using Protocol Buffers is also feasible.The excitement behind PQS is two-fold as alluded. Java-based clients now have two means of running queries: a typical JDBC client with

There's a number of tools from the command line, that I like to use from NiFi as part of a Big Data Flow.The first tool I wanted to use was Ghostscript.On Centos/RHEL, you can install it via:yum install ghostscriptI use GhostScript to extract content from PDFs (this can be passed in from an existing flow using ExecuteStreamCommand). It then outputs to the Standard Output the text from those files. run.sh gs -dBATCH -dNOPAUSE -sDEVICE=txtwrite -dFirstPage=1 -dLastPage=500 -sOutputFile=- $@Output from the Hadoop documentation:GPL Ghostscript 9.07 (2013-02-14)Copyright (C) 2012 Artifex Software, Inc. All rights reserved.This software comes with NO WARRANTY: see the file PUBLIC for details.Processing pages 1 through 8.Page 1Can't find (or can't open) font file NimbusSanL-ReguItal.Querying operating system for font files...Loading NimbusSanL-ReguItal font from /usr/share/fonts/default/Type1/n019023l.pfb... 3984660 2473586 2498328 1197484 3 done.Loading NimbusSanL-Bold

I was working on one of the case and came across an issue where balancer utility was not working in namenode HA environment.HDP Version - 2.3.2.0Ambari Version - 2.1.2Issue - balancer was failing with below error:[hdfs@slave1 ~]$ hdfs balancer 15/11/20 11:46:10 INFO balancer.Balancer: namenodes = [hdfs://mycluster, hdfs://mycluster:8020] 15/11/20 11:46:10 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] [..] 15/11/20 11:46:11 INFO block.BlockTokenSecretManager: Setting block keys 15/11/20 11:46:11 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Nov 20, 2015 11:46:11 AM Balancing took 1.883 seconds Resolution(workaround) -1. Copy /etc/hadoop/conf/* /home/hdfs/copied_conf/ 2. Edit /home/hdfs/copied_conf/hdfs-site.xml and delete below property &lt

When executing Step 3 of the Ambari installation wizard "Confirm Hosts", Ambari will (by default) SSH to each node and start an instance of the Ambari Agent process. In some cases, it is possible that the local RPM database is corrupted and this registration process will fail.The error message in Ambari would look something like:INFO:root:Executing parallel bootstrapERROR:root:ERROR: Bootstrap of host myhost.mydomain fails because previous action finished with non-zero exit code (1)ERROR MESSAGE: tcgetattr: Invalid argumentConnection to myhost.mydomain closed.STDOUT: Error: database disk image is malformedError: database disk image is malformedDesired version (2.5.0.0) of ambari-agent package is not available.tcgetattr: Invalid argumentConnection to myhost.mydomain closed.In this case, the local RPM database is malformed and all actions to alter the installed packages on the system will fail until the database is rebuilt.This can be done by the following commands as root on the host re

After creating a cluster on the HDCLoud for AWS, you may notice that certain ports are not opened by default, so you may need to manually open these ports by editing the inbound access on the security group.In this tutorial, I will show you how to open YARN Resource Manager UI (8088) and Hive UI (10502) ports by manually editing the inbound access on the master node security group.Let’s get started!1. On AWS, from the Services menu, select EC2 to navigate to the EC2 console:2. In the left pane, in the INSTANCES section, click on Instances.Note: If you can’t see your instances, check the top right corner to make sure that you are in the correct region.3. Identify the instance corresponding to your master node and. The name of the instance should be <your-cluster-name>-1-master. Next, select that instance. This will allow you to see the Description tab, which includes the link to the security group configuration:4. Click on the security group URL to open the Security Group section.

Use Case: Store log data in Hadoop Data Lake and Send Curated, Reduced Set to Sumologic via REST APIIntegrationThe integration point for sending log data to Sumologic is their HTTP Source. To send data you must setup an HTTP Source in Sumologic from your web console as shown below. Take the HTTP string they give to you and put into an InvokeHTTP processor with POST, it will look something like this: https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/ZaLongCodeLongI noticed IP Address in the data, so I decided to parse it out:${regex.6:substringAfterLast('source ip: '):replaceAll('\)','')}Then send it to MaxMind for processing. The MaxMind GeoIP free database is easy to download and use with NiFi. Just add the GeoIP processor and connect the field and the file location.Finally displaying and charting data is up next, easy as pie in Zeppelin. Just query my Phoenix data. The flow is a bit long as I am using RegEx to convert the logs from NiFi Log4J format to indiv

If you are on a Mac and run into the following error when running start-all.sh:% sh start-all.shstarting org.apache.spark.deploy.master.Master, logging to ...localhost: ssh: connect to host localhost port 22: Connection refusedYou need to enable "Remote Login" for your machine. From System Preferences, select Sharing, and then turn on Remote Login.

SAS users are very excited about leveraging Hadoop with their existing deployments. This article will cover the basic concepts of SAS Access to Hadoop, and add on product from SAS which can be deployed into existing SAS enviroments. This product give a SAS user the ability to leverage HDFS storage, access HiveServer2 and execute inline within a SAS program, HDFS, Pig, Hive and MapReduce programs. Lets start by discussing how to leverage HDFS storage.1) Leveraging HDFS for flat files using the SAS Filename StatementThe SAS Filename statement allows a SAS programmer to setup a pointer to an inbound or outbound filesystem directory. With Hadoop, the SAS Filename statement can reference an HDFS directory. Once the file reference is established, this file reference can be used within a SAS Data Step on an Infile or File statement. This enables SAS programmers to read and write flat files to and from HDFS inline within their programs.2) Leveraging HDFS for SAS Libraries using the Libna