Resources

Solr indexing the MySQL database table on HDP 2.5 Tech Preview:Solr version used: solr 4.9.0Step1: Downloaded the solr 4.9.0.zip from https://archive.apache.org/dist/lucene/solr/4.9.0/Step2: Extract the file:Step3: modify the solrconfig.xml, schema.xml and add the db-data-config.xml atStep4: add the jar at this location a.vi solrconfig.xml: add these lines in between the config tags.<lib dir="../../../contrib/dataimporthandler/lib/" regex=".*\.jar" /><lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" /><lib dir="../../../lib/" regex="mysql-connector-java-5.0.8-bin.jar" /><requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> <lst name="defaults"> <str name="config">db-data-config.xml</str> </lst></requestHandler> b.vi schema.xml add the below line:<dynamicField name="*_name" type="text_general" multiValued="false" indexed="true" stored="true" /> c.Create a file ca

Since version 2.6, Apache Hadoop has had the ability to encrypt files that are written to special directories called encryption zones. In order for this at-rest encryption to work, encryption keys need to be managed by a Key Management Service (KMS). Apache Ranger 0.5 provided a scalable, open source KMS to provide key management for the Hadoop ecosystem. These features have made it easier to implement business and mission critical applications on Hadoop where security is a concern. These business/mission critical applications have also brought with them the need for fault tolerance and disaster recovery. Using Apache Falcon, it is easy to configure the copying of data from the Production Hadoop cluster to an off-site Disaster Recover (DR) cluster. But what is the best way to handle the encrypted data? Decrypting/encrypting the data to transfer it can hinder performance, but how do you decrypt data on the DR site without the proper keys from the KMS?In this article, we will investigate

Overview This document gives an overview of HDP Installation on Isilon. The pdf version of the article with images - installation-guide-emc-isilon-hdp-23.pdf Architecture In installing Hadoop with Isilon, the key difference is that, each Isilon Node contains a Hadoop Compatible NameNode and DataNode. The compute and the storage are on separate set of node unlike a common of Hadoop Architecture. HDP will connect to the EMC Isilon cluster using EMC SmartConnect. SmartConnect allows for transparent failover in case an Isilon Node Failure. The transactions will seamlessly failover to the new nodes. Isilon has various storage architectures which allows fast recovery of the failed nodes and file level data protection settings. Certification EMC Isilon has 2 steps with their release process Step 1 – Compliance Compatible Step 2 – Certified with HWX certification suite EMC will release its support for Hortonworks HDP, stating it is compliance compatible. There is a few

Most frequently there are questions on how to add/delete groups/users from Ambari. This document describes how to add/delete ambari's local users and groups (Delete applies to LDAP users/groups as well).This document should give a consolidated list of rest calls to manage users/groups.Following are the steps for end-to-end creation/updating users/groups using ambari REST API.1) Creating a user :curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"Users/user_name":"{USER}","Users/password":"{PASSWORD}","Users/active":"{ISACTIVE}","Users/admin":"{ISADMIN}"}' http://ambari-server:8080/api/v1/users2) Change password of user :curl -iv -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"Users/password":"{UPDATED_PASSWORD}"}' http://ambari-server:8080/api/v1/users/{USER}3) Toggle user's admin status:curl -iv -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"Users/admin":"{true/false}"}' http://ambari-server:8080/api/v1/users/{USER}4) Creating a group :curl -iv -u admin:

Authors - Gerdan dos Santos - https://community.hortonworks.com/users/12342/gerdan.html Pedro Drummond - https://community.hortonworks.com/users/10987/pedrodrummond.html After deleting users using REST API, one might get error "404 not found". This articles demonstrates how to automatically delete the "ghost" users from MySQL and Posgres.Reproducing the ErrorCreating testUser1. Create testUser at Ranger UI - http://{ranger_host}:6080 2. Username: testUser; Group: public . 3. Check if the default Resource Based Policies is assigned to testUser . 4. If testUser is not assigned to the policy, add it and click the save button . Deleting testUser using REST API Reference: https://community.hortonworks.com/questions/1021/how-to-remove-all-external-users-from-the-ranger-r.html Through the terminal, delete the testUser using REST API. For instance: curl -X DELETE -v -u admin:password http://{RangerHost}:6080/service/xusers/users/userName/testUser You should get something l

I am a junkie for faster & cheaper data processing. Exactly why I love IaaS. My personal REAL WORLD experience with the typically IaaS providers has been generally slow on performance. Not to say hadoop/hbase/spark/etc jobs will not perform; however, you need to be familiar with what you're getting into and set realistic expectations. Recently I meet the IaaS vendor Their liquid metal offering which provides all the greatness which comes with bare metal on-prem installations but in the cloud. Options for bonded NICs & DAS had me at hello.I decided to run the same performance test I ran on AWS (article here) on bigstep. All the details of the scripts I ran are in that article. Just a quick note - these performance articles do not advocate for or against any specific IaaS provider. Nor does it reflect the HDP software. I simply want to run the repeatable processing test with near/similar IaaS hardware profiles and gather performance statistics. Interrupt the numbers as you

Apache Zeppelin (version 0.6.0) includes the ability to securely authenticate users and require logins. It uses the Apache Shiro security framework to accomplish this objective. Note: prior versions of Zeppelin did not force users to login. After launching the HDP 2.5 Tech Preview Sandbox on a virtual machine, make sure the Zeppelin service is up and running via Ambari. Next, open the Zeppelin UI either by clicking on:Services (tab) -> Zeppelin notebook (left-hand panel) -> Quick Links (tab) -> "Zeppelin UI" (button)or just by opening a browser at: http://sandbox.hortonworks.com:9995/ (or http://127.0.0.1:9995/)The Zeppelin welcome page should show in the browser, and you should notice a "Login" button in the upper right-hand corner. This will bring up a pop-up window with text entries for username and password. Enter one of the username/password pairs below (these are the defaults listed in the "shiro.ini" file located in the "conf" sub-directory of zeppelin):Username/Passwor

This article is a companion to the article "Avro Schema Registry with Apache Atlas for Streaming Data Management". https://community.hortonworks.com/articles/51379/avro-schema-registry-with-apache-atlas-for-streami.htmlThe article explores how an Avro schema registry can bring data governance to streaming data and the benefits that come with it. This tutorial demonstrates the implementation of this concept and some of the resulting features. Download HDP 2.5 Sandboxmodify the hosts file on the local machine to resolve sandbox.hortonworks.com to 127.0.0.1SSH to the Sandbox (ssh root@sandbox.hortonworks.com -p 2222)ambari-admin-password-resetmake sure to set the Ambari password to "admin" Log into Ambari and start the following services (http://sandbox.hortonworks.com:8080/) HBaseLog SearchKafkaAtlasFrom the SSH console: git clone https://github.com/vakshorton/AvroSchemaShreddercd /root/AvroSchemaShredderchmod 755 install.sh./install.shjava -jar AvroSchemaShredder-jar-with-dependencies.

Data is the becoming the new precious resource. As the world produces more and more data, business units find increasingly more ways to monetize that data. This means that data that used to be retained for a short time or not at all, is now being persisted long term. This data is being gathered from more and more sources and not necessarily from within the organization that uses it. It is also increasingly being generated by machines, meaning that the volume, velocity, and variety of the data proliferate at an overwhelming rate. There are now lots of tools that enable an organization to address the challenges imposed by the proliferation of data. However, many organizations have been focused on dealing with volume and velocity while not focusing on the challenges created by the lack or inconsistency of structure. In order to truly unlock the power of all that data, an organization must first apply a consistent set of guidelines for governance of the data. Getting value from new data so

IntroductionThis article is not meant to show how to install or create a “Hello World” Nifi data flow, but how to resolve a data filtering problem with NiFi providing two approaches, using a filter list as a file on the disk, which could be static or dynamic, and a list stored in a distributed cache populated from the same file. The amount of data used was minimal and simplistic and no performance difference can be perceived, however, at scale, where memory is available, a caching implementation should perform better.This article assumes some familiarity with NiFi, knowing what a processor or a queue is and how to set the basic configurations for a processor or a queue, also how to visualize the data at various steps throughout the flow, starting and stopping processors.Since you are somehow familiar with Nifi, you probably know how to install it and start it, however, I will provide a quick refresher below.Pre-requisitesFor this demo, I used the latest version of Nifi available at the

Purpose of this article: When you install HDP for dev/test environment, you would repeat same commands to set up your host OS. To save time, created a BASH script which helps to set up the host OS (Ubuntu only) and docker image (CentOS). What this script does: Install packages on Ubuntu host OS Set up docker, such as creating image and spawning containers [Optional] Set up a local repository for HDP (not Ambari) with Apache2 What this script does NOT: As of this writing, this does not install HDP Please use Ambari Blueprint if you would like to automate HDP installation as well This step is NOT for production environment but would be useful to test HA components Host set up steps: Install Ubuntu 14.x LTS on your VirtualBox/VMware/Azure/AWS. It should be easy to deploy Ubuntu VM if you use Azure or AWS. If you are using VirtualBox/VMWare, you might want to backup Ubuntu installed VM as a template, so that later you can clone. Login to Ubuntu and become root (sudo -

How to setup cross realm trust between two MIT KDC – In this post, we will see how to setup cross realm trust between two MIT KDC. We can access and copy data from one cluster to another if the cross realm trust is setup correctly..In our example, we have 2 clusters with same HDP version(2.4.2.0) and Ambari version(2.2.2.0).Cluster 1:172.26.68.47 hwx-1.hwx.com  hwx-1172.26.68.46 hwx-2.hwx.com  hwx-2172.26.68.45 hwx-3.hwx.com  hwx-3Cluster 2:172.26.68.48 support-1.support.com support-1172.26.68.49 support-2.support.com support-2172.26.68.50 support-3.support.com support-3.Below are the steps:.Step 1: Make sure both the clusters are kerberized with MIT KDC. You can use below automated script for configuring Kerberos on HDP.https://community.hortonworks.com/articles/29203/automated-kerberos-installation-and-configuration.html.Step 2: Please configure /etc/hosts file on both the clusters to have Ip <-> hostname mappings.Example:On both clusters /etc/hosts file should look like

Apache ZooKeeper is a “high-performance coordination service for distributed applications.” Most users do not use ZooKeeper directly; however, most users are also hard-pressed to deploy a Hadoop-based architecture that doesn’t rely on ZooKeeper in some way. With its prevalence in the data-center, resource management within ZooKeeper is paramount to ensure that the various applications and services relying on ZooKeeper are able to access it in a timely manner. To this end, one of ZooKeeper’s protection mechanisms is known as “max client connections” or “ maxClientCnxns”. maxClientCnxns refers to a configuration property that can be added to the zoo.cfg configuration file. This property limits the number of active connections from a host, specified by IP address, to a single ZooKeeper server. By default, this limit is 60 active connections; one host is not allowed to have more than 60 active connections open to one ZooKeeper server. Changes to this property in the zoo.cfg file requir

Kafka from 0.9 onwards started support SASL_PLAINTEXT ( authentication and non-encrypted) for communication b/w brokers and consumer/produce r with broker. To know more about SASL, please refer to this link. Maven Dependency Add below maven dependency to your pom.xml <dependency><br> <groupId>org.apache.kafka</groupId><br> <artifactId>kafka-clients</artifactId><br> <version>0.10.0.0</version><br></dependency> 2. Kerberos Setup Configure JAAS configuration file with contents as belowKafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true principal="user@EXAMPLE.COM" useKeyTab=true serviceName="kafka" keyTab="/etc/security/keytabs/user.headless.keytab";};Above configuration is set to use key tab and ticket cache. The Kakfa Client Producer use above info to get TGT and authenticates with Kafka broker.Note:a] Make sure the /etc/krb5.conf has realms mapping for "EXAMPLE.COM" an

Overview ORC provides many benefits, including support for compression. However, there are times when you need to store “raw” data in HDFS, but want to take advantage of compression. One such use case is with Hive external tables. Using PIG, you can LOAD the contents of uncompressed data from an HDFS directory, and then STORE that data in a compressed format into another HDFS directory. This approach is particularly useful if you are already using PIG for part of your ETL processing.Scope This tutorial has been tested with the following configuration: Mac OS X 10.11.5 VirtualBox 5.1.2 HDP 2.5 Tech Preview on Hortonworks SandboxPrerequisites The following should be completed before following this tutorial: VirtualBox for Mac OS X installed VirtualBox Link Hortonworks HDP 2.5 Technical Preview Sandbox configured in VirtualBox Sandbox Link Steps1. Connect to the Sandbox If you are using Vagrant to spin up your sandbox, then you can simply connect using: $ vagrant ssh If

With Apache NiFi 1.0 you can now act as a simple SMTP server (though it is recommended to sit behind a real SMTP MTA and just get mail forwards). It makes for an easy way to ingest mail, headers and attachments.The first thing you will notice is the awesome new UI, which is much cleaner and a joy to use.'First add a processor, ListenSMTP, this will be your mail gateway/SMTP server. As you can see there's also processors for extracting attachments and headers from Email.You need to make sure you set Listening Port, SMTP hostname and Max. # of Connections.The entire flow for mail processing is pretty simple, but easy to follow. We listen for SMTP over TCP Port (I chose 2025, but with Root access you could run on 25). I send the original flow file right to HDFS. I extract the attachments and put them in a separate HDFS directory and finally pull out the email headers and also send them to an HDFS file. I have a little test flow in the bottom to read a file and send email to o

1. Recover the lease for the fileWhen you do "hdfs dfs -cat file1" from the command line, you get the exception saying that it "Cannot obtain block length for LocatedBlock". Usually this means the file is still in being-written state, i.e., it has not been closed yet, and the reader cannot successfully identify its current length by communicating with corresponding DataNodes.Suppose you're pretty sure the writer client is dead, killed, or lost connection to the servers. You're wondering what else you can do other than waiting.hdfs debug recoverLease -path <path-of-the-file> [-retries <retry-times>] This command will ask the NameNode to try to recover the lease for the file, and based on the NameNode log you may track to detailed DataNodes to understand the states of the replicas. The command may successfully close the file if there are still healthy replicas. Otherwise we can get more internal details about the file/block state. Please refer to https://community.hortonwo

The Apache Nifi community recently released the beta version of Apache Nifi 1.0.0. This version comes with significant updates, which include a UI refresh, transition to zero master clustering, added multi-tenant authorization, and templates that are now deterministically ordered allowing for version controlled templates! The beta also boasts nine new processors, bringing the total to 165. The full list of release notes can be found here. Below I will do a very basic walkthrough of some of the UI changes from 0.7.0 to the 1.0.0 beta: Original Apache Nifi 0.7.0 Flow New Apache Nifi 1.0.0 Beta Outside of the more modernized look and feel, there are some key UI Changes: (1) Apache Nifi 1.0.0 beta now includes a status bar showing statistics of the overall flow, including bytes in and out, number of started and stopped processors, processors in error, last refresh, etc.(2) The beta also has new collapsable navigation and operation panes (3) There's a new drop down menu where you c

Test Conducted on:1) HDP2.3.42) Data Size : 1.4 GB2) Cluster is ideal and not running any other jobs.Conclusion:Observed that Zlib is doing more compression than SNAPPY but SNAPPY jobs are completing quicker than ZLib.Results are here:Execution Time in Sec:Disk Usage Comparison:

Just a few months ago, Apache Storm announced release 1.0 for the distribution. The bullet points below summarize the new features available. For more detailed descriptions, you can go to this link to read the full release notes:http://storm.apache.org/2016/04/12/storm100-released.htmlApache Storm 1.0 ReleaseApache Storm 1.0 is *up to 16 times faster than previous versions, with latency reduced up to 60%.”Pacemaker – Heartbeat ServerPacemaker is an optional Storm daemon designed to process heartbeats from workers. (overcomes scaling problems of Zookeeper)Distributed Cache APIFiles in the distributed cache can be updated at any time from the command line, without the need to redeploy a topology. HA NimbusMultiple instances of the Nimbus service run in a cluster and perform leader election when a Nimbus node failsNative Streaming Window APIStorm has support for sliding and tumbling windows based on time duration and/or event count.Automatic BackpressureStorm now has an automatic backpres

Hortonworks and SAS have partnered to create two new Apache NiFi processors. These processors allow data/events to be streamed between Hortonworks DataFlow (HDF) and SAS Event Stream Processing.Why does this matter? HDF powered by Apache NiFi, Kafka and Storm, is an integrated system for real-time dataflow management and streaming analytics on-premise or in the cloud. SAS Event Stream Processing is a real-time, low-latency, high-throughput event processing solution that can deploy SAS machine learning models. By integrating these technologies, organizations now have the option of deploying their SAS models in real-time within the Hortonworks platform. This offers flexible deployment options for your streaming analytics projects, while providing powerful analytics from SAS. How does this integration work? There are two new processors that can be added to NiFi: ListenESP: This processor initiates a listener within NiFi that receives events from the SAS Event Stream Processi

To enable the SSL/HTTPS for Ambari, have to input signed certificate and Privatekey file. Generally many organizations would have JavaKeyStore/JKS file from which they would have generated a CSR and got the signed certificate from CA. In this case they may not have separate Privatekey file with them. In that scenario below steps can be followed to enable the SSL. Here I am providing the steps to enable the SSL using self signed certificate.1. Create Java key store: keytool -genkey -keyalg RSA -alias ambari -keystore keystore.jks -validity 360 -keysize 2048 2. Export cert to server.cer filekeytool -export -alias ambari -file server.cer -keystore keystore.jks Note: In case of CA signing, CSR can be exported from JKS file and get that signed from appropriate CA.3. Convert .cer file from DER format to PEM formatopenssl x509 -in server.cer -inform der -outform pem -out server.pem 3. Export Privatekey to PKCS12 keytool -v -importkeystore -srckeystore keystore.jks -srcalias ambari -des

UPDATE: I'm happy to report that my patch for PIG-4931 was accepted and merged to trunk.I was browsing through Apache Pig Jiras and stumbled on Jira https://issues.apache.org/jira/browse/PIG-4931 requiring to document Pig "IN" operator. Turns out Pig had IN operator since days of 0.12 and no one had a chance to document it yet. The associated JIRA is https://issues.apache.org/jira/browse/PIG-3269. In this short article I will go over the IN operator and until I'm able to submit a patch to close out the ticket this should serve as its documentation. Now, IN operator in Pig works like in SQL. You provide a list of fields and it will return just those rows. It is a lot more useful than for example a = LOAD '1.txt' USING PigStorage(',') AS (i:int); b = FILTER a BY (i == 1) OR (i == 22) OR (i == 333) OR (i == 4444) OR (i == 55555); You can rewrite the same statement as a = LOAD '1.txt' USING PigStorage(',') AS (i:int); b = FILTER a BY i IN (1,22,333,4444,55555); The best thing

Working with customers I get this question quite often on how to size their cluster. Apart from master nodes, one question that often comes up is the size of data nodes. Hardware vendors now offer disks up to 8TB capacity that can offer customers up to 96 TB of storage per node assuming twelve spindles per node. Even if you go with 12x4TB disks, that is still a whopping 48 TB of storage per node. For most cases, I have always recommended my customers 12x2TB disks over last three years and I continue to do so, given bandwidth remains very expensive, and as we'll see below, it is a very important component when you are sizing a cluster and deciding between high and low density data nodes.The calculations I am sharing here were done for a customer when they told me that re-replication of blocks when a node fails takes a very long time. This customer had 12x4TB disks on each node.So rather than preferring one opinion over the other, let's do some maths and then decide what works for your u

Hive Streaming Compaction This is the second part of the Hive Streaming Article series. In this article we will review the issues around compacting Hive Streaming files. One of the results of ingesting data through Hive streaming is the creation of many small 'Delta' files. Left uncompacted you could run the risk of running into NameNode capacity problems. Fortunately, compaction functionality is part of Hive Streaming. The remainder of this Article reviews design considerations as well as commands necessary to enable and control compaction for your Hive tables. Hive Compaction Design considerations The Compaction process has a set of cleaner processes running in the background during the ingest process looking for opportunities to compact the delta files based on the rules you specify. The first thing to keep in mind is that there are two forms of Compaction; ‘minor’ and ‘major’. A ‘minor’ compaction will just consolidate the delta files. This approach does not have to worr