Community Articles

There are three main installation paths for creating a new Cloudera Manager deployment:   Installation Path A - Automated Installation by Cloudera Manager (Non-Production Mode). "This path is recommended for demonstration and proof-of-concept deployments, but is not recommended for production deployments because it's not intended to scale and may require database migration as your cluster grows"[1].   Installation Path B - Installation Using Cloudera Manager Parcels or Packages. This path requires you to first manually install and configure an external database for the Cloudera Manager Server, Cloudera Management services and any other CDH services that may reside on this database instance (such as Hue, Hive, Oozie etc.).   Installation Path C - Manual Installation Using Cloudera Manager Tarballs. In this path, you install the the Oracle JDK, Cloudera Manager server and Agent software using tarballs.   This tutorial covers the Path B installation of Cloudera Manager

IntroductionAmbari 2.4 introduces the Hue-to-Views Migration tool, which is specifically designed to migrate existing Hue artifacts to an Ambari View.Currently in Hue the hive queries are saved in ‘Saved Queries’ in Hue UI. These queries can be invoked by the user by going to Saved Queries in Hue UI. HUETOAMBARI_MIGRATION would help in migrating these saved Hive artifacts from Hue to Ambari Hive View. Once the migration is done, the Saved Queries can be viewed from Ambari Hive View. Currently HUETOAMBARI_MIGRATION would help in migrating the following artifacts from Hue to Ambari Views.Hive - Saved Queries, Query History Pig - Saved Scripts, Pig JobsThe view would use JDBC to connect to Hue database to obtain the artifacts. Once obtained, these would be stored in the Ambari View Server database.Today, the view migrates the scripts one user at a time. The user to be migrated would be specified during the migration.Note:- This view does not validate scripts. Non-working Hue scripts wil

Here is the steps to enable HTTPS for HBASE Master UI. 1. Add following properties in "Custom hbase-site" hbase.ssl.enabled=truehbase.http.policy=HTTPS_ONLY hadoop.ssl.enabled=true2. In HBASE, there is no direct option to add/configure the keystore files (JKS), it uses HADOOP configurations files. Lets say If HTTPS is already enabled for HDFS Web UI then your host would have already configured with JKS files in ssl-server.xml and ssl-client.xml files, so HBASE also uses same ssl-server.xml files. You can find more details to configure HTTPS for HDFS at https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html3. If you are planning to have different JKS files for HDFS and HBASE then you can copy the the ssl-server.xml to /etc/hbase/conf path and configure the JKS file. Restart HBASE server and server should come fine and should be accessable https://HOSTNAME:16010/ Note:Please note that there is a bug in AMBARI because of that HBASE quick always try to open with HTTP

Recently I had a client ask about how would we go about connecting a windows share to Nifi to HDFS, or if it was even possible. This is how you build a working proof of concept to demo the capabilities!You will need two Servers or Virtual machines. One for windows, one for Hadoop + Nifi. I personally elected to use these two The Sandbox http://hortonworks.com/products/hortonworks-sandbox/A windows VM running win 7 https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/linux/You then need to install nifi on the sandbox, I find this repo to be the easiest to follow. https://github.com/abajwa-hw/ambari-nifi-serviceBe sure the servers can talk to each other directly, I personally used a bridged network connection in virtual box and looked up the IPs on my router's control panel.Next you need to setup a windows share of some format. This can be combined with active directory but I personally just enabled guest accounts and made an account called Nifi_Test. These instruct

SYNOPSIS Which is faster when analyzing data using Spark 1.6.1: HDP with HDFS for storage, or EMR? Testing shows that HDP using HDFS has performance gains over using EMR. HDP/HDFS outperforms EMR by 46% when tested against 1 full day of 37 GB Clickstream (Web) data.HDP/HDFSEMRTime Elapsed3 mins, 29 sec5 mins, 5 sec * See below at end of article for validation and screen prints showing Resource Manager logs HDP Hortonworks Data Platform (HDP) is the industry's only true secure, enterprise-ready open source Apache Hadoop distribution. The Hadoop Distributed File System (HDFS) is a Java-based distributed block storage file system that is used to store all data in HDP. EMR Amazon Elastic MapReduce (Amazon EMR) is a managed Hadoop framework to distribute and process vast amounts of data across dynamically scalable Amazon EC2 instances. S3 is an inexpensive object store that can theoretically scale out infinitely without the limitations inherent to a hierarchical block stora

In the recent weeks I have tested Hadoop on various IaaS providers in hope to find additional performance insights. BigStep blew away my expectation in terms of Hadoop performance on IaaS. I wanted to take the testing a step further. Lets quantify performance measures by adding nodes to a cluster. Even for a small 1TB data set, would 5 nodes perform far greater then 3? I have heard a few times when it comes to small datasets adding more nodes may not have a impact. So this led me to test a 3 node cluster vs a 5 node cluster using 1TB dataset. Does the extra 2 nodes increase performance with processing and IO? Lets find out.Started the testing with dfsio which is a distributed IO benchmark tool. Here are results:From 3 to 5 data nodes IO read performance increased approx. 36%From 3 to 5 data nodes IO write performance increased approx. 49%With 2 additional data nodes a performance IO throughput of 49%! Wish I had more boxes to play with. Can't image where this would take the m

New table region split/merge APINew API in HBase HDP 2.5 allows user to disable/enable automatic region splits and merges. From HBase shell you can run the following commands:Enable region splits hbase> splitormerge_switch 'SPLIT', trueDisable region splitshbase> splitormerge_switch 'SPLIT', falseEnable region mergeshbase> splitormerge_switch 'MERGE', trueDisable region mergeshbase> splitormerge_switch 'MERGE', falseCheck region split switch statushbase> splitormerge_enabled 'SPLIT'Check region merge switch statushbase> splitormerge_enabled 'MERGE'Usage in HBase hbck toolHBase hbck tool can automatically use this API during restore operation if the following command-line argument is specified: -disableSplitAndMerge or tool is run in repair mode. Disabling region splits and merges during repair or diagnostic runs improves tool's ability to diagnose and repair HBase cluster.Usage in table snapshots It is recommended now to disable both: region splits and merges before

Sharing source code I wrote to a prospect that wants to try Phoenix + .NET.It's a simple code that POST the query to Phoenix Server and parse result JSON and prints results on console:using System;using System.Net;using System.Collections.Generic;using Newtonsoft.Json;using System.Text; namespace ConsoleApplication1{ class PhoenixSample { public class FirstFrame { public String offset; public List<List<String>> rows; } public class Result { public FirstFrame firstFrame; public int updateCount; } public class PhoenixResult { public string response { get; set; } public List<Result> results { get; set; } } static void Main(string[] args) { try { string url = "http://192.168.56.203:8765"; string query = "select * from test"; var syncClient = new WebClient

Brandon Wilson has a great article that shows how to use the "CACHE TABLE" cmd in Tableau, however more recent drivers have come out and you can now connect directly to the thriftserver using a spark-sql driver. This is using HDP 2.5 and SimbaSparkOdbc.First pull up a Tableau connection and select the thriftServer. Additionally had to open the virtualbox port 10015. Next if you don't have the driver Tableau will jump you to a page where you can download a spark-sql driver and inside that package chose this driver.Once you establish a valid connection you will see Tableau flag the connects based on the driver. Below you will see the Hive connection from Brandon's article and now the new Spark connection.Next using the CACHE cmd enter the below into Tableau's initial SQL box.Finally check the storage of spark for the warehouse/crimes table in memory. Or any table of your chosing for that matter.Some visuals from Tableau.

Microsoft HDInsights provides the four different options under Cluster Type and they are: Hadoop, Hbase, Storm and Spark (Preview); Some of HDP components are not standard in the HDInsight distribution of choice. In this particular case, the customer was interested in Spark / Spark Notebooks Components and especially Zeppelin. When choosing the Spark Option, only these services will be provided under Ambari: HDFS, MR2, YARN, Tez, Hive, Pig, Sqoop, OOzie, Zookeeper, Ambari Metrics, Jupyter, Livy (Livy is the remote job submission server for Spark), and Spark; Zeppelin is missing from the HDI Spark distro at the time of writing this article; So in order to install Zeppelin When choosing 'Spark' as the HDI ‘Cluster Type' , and under 'New HDInsights Cluster' >> 'Optional Configuration’ >> 'Scripts Action' put the following URI as shown below: https://hdiconfigactions.blob.core.windows.net/linuxincubatorzeppelinv01/install-zeppelin-spark151-v01.sh Further services

Linkedin PostHave you heard of computing on traditional disk-based or flash-based technologies? We all use disks/flash storage in our laptops, desktop or servers sitting in the data center.What is Apache Ignite In-Memory Data Fabric ?It's high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time "orders of magnitude faster than possible with traditional disk-based or flash-based technologies."Fabric: In information technology, fabric is a synonym for the words framework or platform. You can view Ignite as a collection of independent, well-integrated, in-memory components geared to improve performance and scalability of your application.Source How Ignite fits with HDFS? (Ignite file system , IGFS)IGFS shakes hand with HDFS. Hadoop can run over IGFS in plug-n-play fashion and significantly reduce I/O and improve both, latency and throughput.Why do we need another layer on the top of HDFS? IGFS supports dual-mode. A

Atlas API : Create Trait TypeTraits are Atlas types that act like decorators to provide a mechanism for classification. Traits can be applied to any Atlas entities (instances of classes). An entity can have any number of traits. For example, a SECURITY_CLEARANCE trait could be defined with an attribute ‘security_level’. An Atlas entity (e.g. an instance of Person class) could be tagged with the SECURITY_CLEARANCE trait by providing a value for the trait’s ‘security_level’ attribute.In order to use a trait, you first need to define the trait type. You can do that by posting to the Types resource of the Atlas REST APIs.Types ResourceA type is the description of any representable item. In this case we are creating a trait type but the same resource is used to represent enums, structs and classes as well. /typesPOSTSubmits a types definition containing a type definition representing a trait.Request BodyTypes Definition application/json Response BodyTypes Response application/json

Spark SQL comes with a nice feature called: "JDBC to other Databases", but, it practice, it's JDBC federation feature."It can be used to create data frames from jdbc databases using scala/python, but it also works directly with Spark SQL Thrift server and allow us to query external JDBC tables seamless like other hive/spark tables."Example below using sandbox 2.3.2 and spark 1.5.1 TP (https://hortonworks.com/hadoop-tutorial/apache-spark-1-5-1-technical-preview-with-hdp-2-3/).This feature works with spark-submit, spark-shell, zeppelin, spark-sql client and spark sql thift server. In this post two examples: #1 using spark-sql thrift server, #2 using spark-shell.Example #1 using Spark SQL Thrift Server1- Run Spark SQL Thrift Server with mysql jdbc driver:sudo -u spark /usr/hdp/2.3.2.1-12/spark/sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10010 --jars "/usr/share/java/mysql-connector-java.jar"2- Open beeline and connect to Spark SQL Thrift Server:beeline -u "jdbc:hive2://

IntroductionWe know that parameter passing is valuable for pig script reuse. One lesser known understanding is that parameters do not simply pass variables to pig scripts but rather (and more fundamentally) they pass text that replaces placeholders in the script. This is a subtle but powerful difference: it means that we can dynamically pass code alternatives to the script at run-time. This allows us to build a script with the same larger purpose but whose logic, settings, schema, storage types, UDFs, etc. can be swapped at runtime. This results in significantly fewer yet more flexible scripts that need to be built and maintained across your projects, group or organization.In this article I will show techniques to leverage this approach through a few script examples. Keep in mind the goal is not the final examples themselves, but rather the possibilities for your own scripting.Key ideas and techniques are:parameters as text substitution-param whose parameter value holds a code sni

Here is an example of scheduling Oozie co-ordinator based on input data events. it starts Oozie workflow when input data is available.In this example coordinator will start at 2016-04-10, 6:00 GMT and will keep running till 2017-02-26, 23:25GMT (please note start and end time in xml file) start="2016-04-10T06:00Z" end="2017-02-26T23:25Z" timezone="GMT"Frequency is 1 day frequency="${coord:days(1)}"Below ETL function gives same value as start time which means coordinator will look for input data which has value same as start data in /user/root/output/YYYYMMDD format <instance>${coord:current(0)}</instance>Below are the working configuration files.coordinator.xml:<coordinator-app name="test" frequency="${coord:days(1)}" start="2016-04-10T06:00Z" end="2017-02-26T23:25Z" timezone="GMT" xmlns="uri:oozie:coordinator:0.2"> <datasets> <dataset name="inputdataset" frequency="${coord:days(1)}" initial-instance="2016-04-10T06:00Z" timezone

Use Case: Process a Media Feed, Store Everything, Run Sentiment Analysis on the Stream, and Act on a ConditionI have my GetTwitter processor looking at my twitter handle and the keyword Hadoop, something I tend to tweet frequently.I use an EvaluateJsonPath to pull out all the attributes I like (msg, user name, geo information, etc...). I use a AttributesToJSON processor to make a new JSON file from just my attributes for a smaller tweet. I store the raw JSON data in HDFS as well in a separate directory.Sending HTML Email is a bit tricky, you need to make sure you don't include extra text, so nothing in the message but RAW HTML as seen below and don't Attach Files or Include All Attributes in Message. Make sure you set the Content Type to text/html.For sentiment analysis I wanted to run something easy, so I use an ExecuteStreamCommand to run a Python 2.7 script that uses NLTK Vader SentimentIntensityAnalyzer. The NiFi part is easy, just a command and call a shell script. The

hdfs dfsadmin -setBalancerBandwidth 100000000 on all the DN and the client we ran the command below hdfs balancer -Dfs.defaultFS=hdfs://<NN_HOSTNAME>:8020 -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.max.concurrent.moves=5 -Ddfs.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5 This will faster balance your HDFS data between datanodes and do this when the cluster is not heavily used. Hope this helps you.

Solr indexing the MySQL database table on HDP 2.5 Tech Preview:Solr version used: solr 4.9.0Step1: Downloaded the solr 4.9.0.zip from https://archive.apache.org/dist/lucene/solr/4.9.0/Step2: Extract the file:Step3: modify the solrconfig.xml, schema.xml and add the db-data-config.xml atStep4: add the jar at this location a.vi solrconfig.xml: add these lines in between the config tags.<lib dir="../../../contrib/dataimporthandler/lib/" regex=".*\.jar" /><lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" /><lib dir="../../../lib/" regex="mysql-connector-java-5.0.8-bin.jar" /><requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler"> <lst name="defaults"> <str name="config">db-data-config.xml</str> </lst></requestHandler> b.vi schema.xml add the below line:<dynamicField name="*_name" type="text_general" multiValued="false" indexed="true" stored="true" /> c.Create a file ca

Since version 2.6, Apache Hadoop has had the ability to encrypt files that are written to special directories called encryption zones. In order for this at-rest encryption to work, encryption keys need to be managed by a Key Management Service (KMS). Apache Ranger 0.5 provided a scalable, open source KMS to provide key management for the Hadoop ecosystem. These features have made it easier to implement business and mission critical applications on Hadoop where security is a concern. These business/mission critical applications have also brought with them the need for fault tolerance and disaster recovery. Using Apache Falcon, it is easy to configure the copying of data from the Production Hadoop cluster to an off-site Disaster Recover (DR) cluster. But what is the best way to handle the encrypted data? Decrypting/encrypting the data to transfer it can hinder performance, but how do you decrypt data on the DR site without the proper keys from the KMS?In this article, we will investigate

Overview This document gives an overview of HDP Installation on Isilon. The pdf version of the article with images - installation-guide-emc-isilon-hdp-23.pdf Architecture In installing Hadoop with Isilon, the key difference is that, each Isilon Node contains a Hadoop Compatible NameNode and DataNode. The compute and the storage are on separate set of node unlike a common of Hadoop Architecture. HDP will connect to the EMC Isilon cluster using EMC SmartConnect. SmartConnect allows for transparent failover in case an Isilon Node Failure. The transactions will seamlessly failover to the new nodes. Isilon has various storage architectures which allows fast recovery of the failed nodes and file level data protection settings. Certification EMC Isilon has 2 steps with their release process Step 1 – Compliance Compatible Step 2 – Certified with HWX certification suite EMC will release its support for Hortonworks HDP, stating it is compliance compatible. There is a few

Most frequently there are questions on how to add/delete groups/users from Ambari. This document describes how to add/delete ambari's local users and groups (Delete applies to LDAP users/groups as well).This document should give a consolidated list of rest calls to manage users/groups.Following are the steps for end-to-end creation/updating users/groups using ambari REST API.1) Creating a user :curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"Users/user_name":"{USER}","Users/password":"{PASSWORD}","Users/active":"{ISACTIVE}","Users/admin":"{ISADMIN}"}' http://ambari-server:8080/api/v1/users2) Change password of user :curl -iv -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"Users/password":"{UPDATED_PASSWORD}"}' http://ambari-server:8080/api/v1/users/{USER}3) Toggle user's admin status:curl -iv -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"Users/admin":"{true/false}"}' http://ambari-server:8080/api/v1/users/{USER}4) Creating a group :curl -iv -u admin:

Authors - Gerdan dos Santos - https://community.hortonworks.com/users/12342/gerdan.html Pedro Drummond - https://community.hortonworks.com/users/10987/pedrodrummond.html After deleting users using REST API, one might get error "404 not found". This articles demonstrates how to automatically delete the "ghost" users from MySQL and Posgres.Reproducing the ErrorCreating testUser1. Create testUser at Ranger UI - http://{ranger_host}:6080 2. Username: testUser; Group: public . 3. Check if the default Resource Based Policies is assigned to testUser . 4. If testUser is not assigned to the policy, add it and click the save button . Deleting testUser using REST API Reference: https://community.hortonworks.com/questions/1021/how-to-remove-all-external-users-from-the-ranger-r.html Through the terminal, delete the testUser using REST API. For instance: curl -X DELETE -v -u admin:password http://{RangerHost}:6080/service/xusers/users/userName/testUser You should get something l

I am a junkie for faster & cheaper data processing. Exactly why I love IaaS. My personal REAL WORLD experience with the typically IaaS providers has been generally slow on performance. Not to say hadoop/hbase/spark/etc jobs will not perform; however, you need to be familiar with what you're getting into and set realistic expectations. Recently I meet the IaaS vendor Their liquid metal offering which provides all the greatness which comes with bare metal on-prem installations but in the cloud. Options for bonded NICs & DAS had me at hello.I decided to run the same performance test I ran on AWS (article here) on bigstep. All the details of the scripts I ran are in that article. Just a quick note - these performance articles do not advocate for or against any specific IaaS provider. Nor does it reflect the HDP software. I simply want to run the repeatable processing test with near/similar IaaS hardware profiles and gather performance statistics. Interrupt the numbers as you

Apache Zeppelin (version 0.6.0) includes the ability to securely authenticate users and require logins. It uses the Apache Shiro security framework to accomplish this objective. Note: prior versions of Zeppelin did not force users to login. After launching the HDP 2.5 Tech Preview Sandbox on a virtual machine, make sure the Zeppelin service is up and running via Ambari. Next, open the Zeppelin UI either by clicking on:Services (tab) -> Zeppelin notebook (left-hand panel) -> Quick Links (tab) -> "Zeppelin UI" (button)or just by opening a browser at: http://sandbox.hortonworks.com:9995/ (or http://127.0.0.1:9995/)The Zeppelin welcome page should show in the browser, and you should notice a "Login" button in the upper right-hand corner. This will bring up a pop-up window with text entries for username and password. Enter one of the username/password pairs below (these are the defaults listed in the "shiro.ini" file located in the "conf" sub-directory of zeppelin):Username/Passwor

This article is a companion to the article "Avro Schema Registry with Apache Atlas for Streaming Data Management". https://community.hortonworks.com/articles/51379/avro-schema-registry-with-apache-atlas-for-streami.htmlThe article explores how an Avro schema registry can bring data governance to streaming data and the benefits that come with it. This tutorial demonstrates the implementation of this concept and some of the resulting features. Download HDP 2.5 Sandboxmodify the hosts file on the local machine to resolve sandbox.hortonworks.com to 127.0.0.1SSH to the Sandbox (ssh root@sandbox.hortonworks.com -p 2222)ambari-admin-password-resetmake sure to set the Ambari password to "admin" Log into Ambari and start the following services (http://sandbox.hortonworks.com:8080/) HBaseLog SearchKafkaAtlasFrom the SSH console: git clone https://github.com/vakshorton/AvroSchemaShreddercd /root/AvroSchemaShredderchmod 755 install.sh./install.shjava -jar AvroSchemaShredder-jar-with-dependencies.