Community Articles

Lipstick InstallationResources:http://www.graphviz.org/Download_linux_rhel.phphttps://github.com/Netflix/Lipstick/wiki/Getting-StartedCommandssudo yum list available 'graphviz*'sudo yum -y install 'graphviz*'./gradlew assembleI always like to rename gradlew, avengers; then./gradlew run-appHit your browser to view:http://localhost:9292/Make sure you add that port/open firewall/etc...2016-06-17 02:36:44,558 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:HadoopVersionPigVersionUserIdStartedAtFinishedAtFeatures2.4.0root2016-06-17 02:36:402016-06-17 02:36:44HASH_JOIN,FILTER,LIMITSuccess!Job Stats (time in seconds):JobIdMapsReducesMaxMapTimeMinMapTImeAvgMapTimeMedianMapTimeMaxReduceTimeMinReduceTimeAvgReduceTimeMedianReducetimeAliasFeatureOutputsjob_local2036219587_000121n/an/an/an/an/an/an/an/afruit_names_join,fruits,limited,namesHASH_JOINjob_local406327028_000211n/an/an/an/an/an/an/an/afruit_namesfile:/tmp/temp195796189/tmp-2027262369,Input(s):Succe

While using Order By operator in Pig Latin,If we have not mentioned ASC or DESC , by default it sorts by ASC order.ex- grunt >OrderedData = order studentDetails by age;This will sort by default in ASC order.Otherwise We can mention here as below:OrderedData = order studentDetails by age DESC;OrderedData = order studentDetails by age ASC;

Sometimes it's easy to share files:https://forums.virtualbox.org/viewtopic.php?t=15679You just pick a directory, set Auto-mount "Yes" and Access to "Full" and hit OK.For some of us, depending on versions of VirtualBox, the VM and host operating system; things might not work. It also can get broken when host operating system or VM updates.Log into the VM as root and try this (if it's HDP sandbox or another running Centos).cd /opt/VBoxGuestAdditions-*/init sudo ./vboxadd setupmodprobe -a vboxguest vboxsf vboxvideorm -rf /media/sf_Downloadsmkdir /media/sf_Downloadsmount -t vboxsf Downloads /media/sf_DownloadsFor me that worked and my Downloads directory was shared so I could move files to my Sandbox and off for development.There are some other things you can try and certainly rebooting everyone helps. For me, this worked fine.

First of all, you need a Hadoop user with correct permission with Ranger (I will use mine : mlanciau), some HDFS quota (if configured), and some YARN queue (I hope it is configured)For this example, I am using default Jenkins configuration with SSH plugin and the TeraSort benchmark suite.1. Configure SSH remote hosts in Jenkins by entering your credential (of course you can use different config)2. Create a new item (on the main portal), I called it Hadoop performance, then click on it and configure3. Click on "This project is parameterised" and add these 2 parameters (you may want to change the parameter ;-))4. Go to build and click on add a step to the build and choose Execute shell script on remote host using ssh5. Then use below information to configure the different scripts (you may want to configure your own user) hdfs dfs -rm -r -skipTrash /user/mlanciau/teragen ; echo 'deleting directory'hdfs dfs -rm -r -skipTrash /user/mlanciau/terasort ; echo 'deleting directory'hdfs dfs -rm -

A customer recently asked about the security capabilities of HIVE with regards to masking and encrypting specific fields within a HIVE table. After some research, I found that there are 4 such UDFs already available in the HIVE 1.3 release. Unfortunately, the customer was using HDP 2.3.0, which ships with HIVE 0.14, and thus they did NOT have access to these UDFs.https://cwiki.apache.org/confluence/display/Hive/L...Being the impatient type, and not wanting to reinvent the wheel, I created a simple github project to back-port these UDFs into, which can be found here: https://github.com/davidkj69/Backported-UDFsThen, I harvested the code from the Apache HIVE 1.3 trunk and added it to the repo, changed the library versions to Hive 0.14, and ran mvn build to generate the Backported-UDFs-0.0.1.jar, which I loaded onto HDFS in the /user/davidk directory. I then was able to test all of the functions as follows: hive> add jar Backported-UDFs-0.0.1.jar; hive> CREATE function sha1 AS 'org

Most code for current big data projects and for the code you are going to write is going to be JVM based (Java and Scala mostly). There is certainly a ton of R, Python, Shell and other languages. For this tutorial we will focus on JVM tools.The great thing about that is that Java and Scala Static Code Analysis Tools will work for analyzing your code. JUnit test are great for testing the basic code and making sure you isolate out functionality from Hadoop and Spark specific interfacing. General Java Tools for Testing http://junit.org/ http://checkstyle.sourceforge.net/http://pmd.github.io/pmd-5.4.2/pmd-java/rules/index.htmlTesting Hadoop (A Great Overview) https://github.com/mfjohnson/HadoopTestinghttps://www.infoq.com/articles/HadoopMRUnitExample:I have a Hive UDF written in Java that I can test via Junit to ensure that the main functionality works.(See: UtilTest)import static org.junit.Assert.assertEquals;import org.junit.Test; /** * Test method for * {@link com.data

H100 Unable to submit statement show databases like '*': org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe"The Ambari view throws the following error: H100 Unable to submit statement show databases like '*': org.apache.thrift.transport.TTransportException:java.net.SocketTimeoutException: Read timed out And I can no longer view database or run queries in the Ambari view. The problem appears to resolve itself after some time, but then will reappear after running a sequence of queries in the Hive view.1) This should be fixed in Ambari 2.22) Browser Refresh fixed it: When connection was open for too long then there was a broken pipe when you tried to access the hive view. When you did a restart to opened a fresh connection and was able to connect.

When working with HDP 2.3.2 sandbox... and running standard sqoop job which moves data from MySQL to Hive.. I ran into the following error:15/11/25 17:00:30 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@135b2072 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@135b2072 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.This was overcome by adding --driver com.mysql.jdbc.Driver to the end of my sqoop create job command, as such:sqoop job -create job1 --import --verbose --connect 'jdbc:mysql://

I was going to just do a REST call to the web service used in my NiFi. My example is on github with full scripts an source code. So I created a semi-useful quick prototype Hive UDF in Java called ProfanityRemover that converts many non-business friendly terms into asterisks (*). It's a small list for performance purposes (like 2,000 with some variations for spacing), but blocks the common ones. It does have a higher than you would like incidence of false positives. To this right you could use a commercial API or write some machine learning. Warning! src/main/resources and src/test/resources in github contain a list of offensive words. Building a Hive UDF To Build an Eclipse Project mvn eclipse:eclipse To Build ./build.sh To Build for Command-Line Usage (outside of Hive) ./buildfirst.sh (or) mvn clean compile assembly:single generates target/deprofaner-1.0-jar-with-dependencies.jar Copy deprofaner*jar to directory to run from or /usr/hdp/current/hive-client/lib/ mkdi

GoalsSetup HDB 2.0 on HDP 2.4.0.0 SandboxAccess HDB 2.0 via pgAdmin3 for interactive accessNotesThis effort is to get up and running with HDB 2.0 on Hortonworks 2.4.0.0 Tryout Sandbox. This steps mentioned here are not intended for production usage and should be merely used as reference.HDB, a.k.a HAWQ will eventually be integrated as a service similar to other addons like Hive, Hbase etceteraThis setup was completed using HDP 2.4.0.0 sandbox which can be downloaded hereArticle assumes that Sandbox is up and running on VMWare Fusion or Virtual boxReference for this article https://cwiki.apache.org/confluence/display/HAWQ/Build+and+InstallFor the purpose of convenience, we will use HAWQ as the term for HDB 2.0Installing HAWQ on HDP 2.4.0.0 HAWQ 2.0 is one of the latest release from Pivotal and can be configured in other version of HDP 2.x as reflected in the articles here:https://community.hortonworks.com/content/kbentry/20420/install-apache-hawq-on-hdp-234.html https://community.horton

In this example we will see how to use a self-signed certificate for initial trials to setup ambari console access over HTTPS. Remember that the self signed certificates are not suitable for production environments. So if your organization does not provide you with an SSL server certificate, you must request a new certificate that is signed by a CA. Users can use several methods to obtain a new signed certificate. For example, users can use Microsoft Internet Information Services (IIS) Manager to request an SSL server certificate from a CA. For testing purposes, you can obtain a free temporary certificate based on an untrusted root from many CAs. To know more about it refer to : https://en.wikipedia.org/wiki/Certificate_authority. This article is divided into 4 sections. Section-1). Generating keys self signed certificates. Section-2). Configuring Ambari Server to use the certificates. Section-3). Common troubleshooting steps.Section-4). Setting up the Ambari Server t

HAWQ is the Greenplum Database/Warehouse engine implemented on top of Hadoop Filesystem. To know GreenplumIt gives an edge to understand how relations are stored within Postgres OR Greenplum database (since Greenplum is based on Postgres, of course with certain optimizations to make it work in parallel and have a master/slave architecture). Unlike hive, a schema is an object within a database. In HAWQs world, this is the setup. HAWQ Cluster - Combination of HAWQ Master, Standby Master and Segment Hosts. All these machines are physical MASTER: Is a postgres master process, linux process, that can share the space with name node depending on the configuration and capacity of the Namenode. STANDBY: Is a replica of MASTER process which includes the logs and metadata information, transferred via log shipping ensuring near realtime replication. Can share the space with Standby/Secondary Namenode host. SEGMENT: Also known as compute nodes, are physical hosts which can host one or m

Atlas Quickstart creates a number of Tags. You may also have created some tags with the REST API. You may want to list the definition of a single Tag or Trait, or you may want a list of all Tags/Traits in Atlas. The following command will list all TRAITS or Tagscurl -iv -d -H "Content-Type: application/json" -X GET http://sandbox.hortonworks.com:21000/api/atlas/types?type=TRAITThe following response shows that I have seven Traits/Tags defined:{"results":["Dimension","ETL","Fact","JdbcAccess","Metric","PII","EXPIRES_ON"],"count":7,"requestId":"qtp1770708318-84 - 6efad306-cb19-4d12-8fd4-31f664e771eb"}The following command returns the definition of a Tag/Trait named, EXPIRES_ON:curl -iv -d -H "Content-Type: application/json" -X GET http://sandbox.hortonworks.com:21000/api/atlas/types/EXPIRES_ONFollowing is the response:{"typeName":"EXPIRES_ON","definition":"{\n \"enumTypes\":[\n \n ],\n \"structTypes\":[\n \n ],\n \"traitTypes\":[\n {\n \"superTypes\":[\n \n

Hi all, here a few steps to get a quick example of around sqooping some Oracle Data into HDFS and Hive table working using the Oracle Developer VM and Hortonworks Sandbox. Very simple but may provide some help for people just starting out. I will be using VirtualBox for this walkthrough so I assuming you already have this installed. Also the Oracle VM will require about 2 Gig of memory and Sandbox about 8 Gig, so you will need a machine with decent a amount of memory to give this a try. I used a mac with 16 gig and it ran fine.Step1: Download Hortonworks Sandbox and import ova into VirtualBoxhttp://hortonworks.com/products/sandbox/Step2: Download Oracle Developer VM (this may require setting up an Oracle free account) and import ova into virtualbox.http://www.oracle.com/technetwork/database/enterprise-edition/databaseappdev-vm-161299.htmlStep3: Set up the 2 VMs so they can communicate with each other. There are many options here, but I setup a Nat Network on the second network ad

Prerequisites: HDP 2.xOneFS 7.2Ambari should report all green for all hosts in clusterForward and reverse hostnames or smart connect should be configuredKerberos Requirements: All KDC’s need to have different realm nameOne KDC per zoneDisable AES encryption in client krb5.confDeleting principals from Isilon doesn’t remove them from kdcDon’t use the isi auth krb5 spn fix commandOverview:Following these steps in the order below will accomplish these tasks: KDC Setup: install and configureHadoop Client Setup: Kerberos configured and testedSecure Isilon Setup: configure, create principals and set proxyusersFinish Hadoop Client Setup: Create all necessary principals, place keytabs on the correct hosts and start servicesFinish Hadoop Client Setup: kerberos_only configurationKDC Setup:Configure KDC: http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-1-2.html below is an overview of the steps taken in the link providedFollow section 13:

we should follow the given steps for remote debugging of the ranger:1. clone the incubator ranger code to your local machine & Import the Apache Ranger project into eclipse.http://ranger.apache.org/quick_start_guide.html2. Now lets say you want to debug the Ranger Admin process then go to the machine where Ranger Admin process is running.edit the /usr/bin/ranger-admin and add the entry as follows:JAVA_OPTS="$JAVA_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9000"keep suspend to n , if nothing need to be debug in startup flow.keep suspend to y, if ranger startup flow need to be debug. in this case process will wait for eclipse to connect to the debug port before it starts3. check whether debug port 9000 is open for ranger admin process netstat -tanlp | grep 16776tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 16776/java4. go to eclipse and in your project Debug as -> Debug Configurations -> Remote Java Applicatio

Here is an example of constructing a complex filter in HBase using Stargate REST APIConsider a Sample Input Table called 'datatable' with the following data:hbase(main):026:0> scan 'datatable'ROW COLUMN+CELLkey1 column=cf1:col1, timestamp=1465135446446, value=value1key1 column=cf1:col2, timestamp=1465135452934, value=value1key2 column=cf2:col1, timestamp=1465135471613, value=value2key2 column=cf2:col2, timestamp=1465135481068, value=value2key3 column=cf1:col1, timestamp=1465137223212, value=value3key3 column=cf2:col2, timestamp=1465137232291, value=value33 row(s) in 0.0300 secondsRequirementFilter the rows that matches any one of the following conditions:Row key is "key1", where Column is {cf1:col1} (OR) Column

I will add the NiFi flow information here tomorrow.For a rough draft, I wanted to show what it could do as it's pretty cool.cat all.txt| jq --raw-output '.["text"]' | syntaxnet/demo.shFrom NiFi I collect a stream of Twitter data and send that to a file as JSON (all.txt). There are many ways to parse that, but I am a fan of the simple command line tool JQ which is an awesome tool to parse JSON from the command line and is available for MacOSX and Linux. So from the Twitter feed I just grab the tweet text to parse with Parsey.Initially I was going to install TensorFlow Syntaxnet (Parsey McParseface) on the HDP 2.4 Sandbox, but Centos 6 and TensorFlow do not play well. So for now the easiest route is to install HDF on a Mac and build Syntaxnet on your Mac.The install instructions are very detailed, but the build is very particular and very machine intensive. It's best to let the build run and go off and do something else with everything else shutdown (no Chrome, VM, editors, ...).Aft

Quite often, hbase users face the following exception when submitting job(s) to secure hbase cluster:2016-05-27 21:16:06,806 WARN [hconnection-0x6be9ad6c-metaLookup-shared--pool2-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2016-05-27 21:16:06,806 FATAL [hconnection-0x6be9ad6c-metaLookup-shared--pool2-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect

This part 1 article is an introduction to Burrow’s design, an overview of the features meant to address some of the known Kafka’s monitoring challenges, a step-by-step tutorial to setup Burrow on sandbox. The part 2 article will explore Consumer lag evaluation rules, HTTP endpoint APIs, email and HTTP notifiers. Stay tuned for part 2 article to be released on May.The source reference is Burrow’s documentation wiki: https://github.com/linkedin/Burrow/wikiWhy Monitoring Kafka? Collecting metrics can help troubleshooting and set alerts on those that require an action.What to Monitor? Process state, memory usage, swap usage, network bandwidth, disk usage, disk IO, under-replicated partitions, offline partitions, active controller brokers, incoming messages per second, incoming/outgoing bytes per second, requests per second, total and split time to process a request, disputed leader elections rate, asynchronous disk log flush and time, unclean leader election rate, number of partitions, I

I call twitter with a filter on some terms, grab the key twitter attributes then call a filter to remove profanities.http://www.purgomalum.com/service/plain?text= or http://www.purgomalum.com/service/json=text work and are free REST API services.For fun, I send the tweet as a search keyword to Guardian using their API (you need to register for a key).http://content.guardianapis.com/search?order-by=newest&q=${tweet}&api-key=StuffNumbers

A remote Linux system can use NFS (Network File System) to mount an HDFS file system and interact with the file system. Before proceeding, it's important to understand that your linux instance is directly accessing your HDFS system through the network, therefore you will incur network latency. Depending on your dataset size, you have to remember you could be potentially processing gigabytes or more of data on a single machine therefore this is not the best approach for large datasets. These steps will show you how to mount and interact with a remote HDFS node within your Linux system:1) The linux system must have NFS installed (CentOS for demo)yum install nfs-utils nfs-utils-lib2) Your HDP cluster must have an NFS Gateway installed (Ambari allows this option with one click)* Keep track of either the FQDN or IP address of the NFSGateway3) In Ambari, under HDFS > Advanced > General set Access time precision = 36000003) Mount the NFS Gateway on your linux system (must be root)moun

EMC Blog Post containing instructions when upgrading HDP 2.2 to HDP 2.3 with IsilonWe have been working with the Isilon engineering team to document the upgrade process from HDP 2.2. to HDP 2.3 when the underlying storage layer is OneFS instead of HDFS. The instructions in the blog post are to be used in conjunction with Ambari instructions on upgrading HDP.Important to note that Isilon is working on making enhancements to OneFS to support rolling upgrade. This will be made available in an upcoming release of OneFS.

Predict Stock Portfolio Gains Using Monte CarloWhy? Why create yet another VaR example? To demonstrate VaR running on a modern architecture that has no vertical limit. This is a functional, immutable, scaleable interpretation of a basic technique commonly used in finance. Code Available here and on github. https://github.com/kirkhas/zeppelin-notebooks/ link to Vlad's article for history of Monte Carlo and VaR - https://community.hortonworks.com/articles/36321/predicting-stock-portfolio-losses-using-monte-carl.htmlSome modifications from original posting include: scala calling Yahoo API directly, alleviating the need for shell scripting and adding interopability between variables. All data loaded dynamically in memory, removing the need to store files (which inherently adds manual customizations to a generic process). Code all in Zeppelin for readability. Visualizations in Zeppelin. Inputs built in using Zep forms so the user can interact with the model. Percentiles not only o

This beautiful demo is courtesy of Vadim Vaks. It utilizes a Lambda architecture built using the Hortonworks Data Platform and Hortonworks Data Flow. The demo shows how a Telecom can manage customer device outages using predictive maintenance and a connected workforce.Overview:Customer devices that are simulated in this specific telecom usecase are SetTopBoxes(STB) in individual homes that might need assistance from a technician, when something goes wrong. Attributes associated with STB are:1. SignalStrength2. Internal Temperature3. StatusLocation of technician is tracked and plotted on MapUI using the latitude and longitudes.Two cycles of operation of STBs are:1. Normal Cycle: When status is normal and internal temperature of STB fluctuates up and down2. Failure Cycle: When status is not normal and internal temperature of STBox incrementally goes high until 109 degrees. Step 1: Prerequisites for the Demo Set Up: For instructions to install this demo on an HDP 2.4 Sandbox,a good p