Community Articles

In the following video, I demonstrate how to create a role-based Cloudbreak credential on AWS Video link: https://youtu.be/XlvUahqwThI The Cloudbreak version used in this video is Cloudbreak 2.7.1. To obtain the policy that need to be assigned to the user, refer to Create CredentialRole documentation.If you are using a corporate AWS account you may be unable to perform some of the steps (such as role creation and policy assignment) by yourself and you may have to contact your AWS admin to perform these steps for you.If you are using a Cloudbreak version different than 2.7.1, refer to the equivalent documentation for that version. You can access Cloudbreak documentation from the Hortonworks docs page at https://docs.hortonworks.com.

There are at times we would need to move kerberos database to different nodes or upgrade the OS of KDC node ( for e.x CentOS6 to CentOS7 ). Obviously you would not want to lose you the kdc users especially if your HDP cluster is configured to use this kdc.Follow below steps to backup and restore kerberos database.prerequisite* Backup the keytab from the HDP cluster under /etc/security/keytabs from all nodes.* Note down your kdc admin principal and password* Backup /etc/krb5.conf* Backup /var/kerberos directoryBackup* Take the kerberos database dump using below command ( to be executed on node running kerberos )kdb5_util dump kdb5_dump.txt* Safely backup the kdb5_dump.txt.Restore* Restore the kerberos database execute below commandkdb5_util load kdb5_dump.txt * Restore the /etc/krb5.conf from backup* Restore /var/kerberos/krb5kdc/kdc.conf from backup* Restore /var/kerberos/krb5kdc/kadm5.acl from backup* Run below command to store master principal in stash file ( kdc admin password is r

For HDFS analysis, an Activity Analyzer needs to be deployed to each NameNode in the cluster. These instances will automatically begin processing the fsimage on startup and will reprocess the latest fsimage data once every 24 hours. By default, when deployed on a NameNode, these Activity Analyzers do not process YARN, or MapReduce & Tez utilization data; This is to reduce the amount of processing done on servers hosting critical services like the NameNode.Unless we install analyzer on namenode, it will not be able to find fsimage in directory and hence fail with below error2018-08-23 14:28:04,401 ERROR [pool-3-thread-1] ActivityManager:78 - Failed to process activity id null of type HDFSjava.lang.NullPointerException at java.io.File.<init>(File.java:277) at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:71) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:69) at

Overview As mentioned in the article parent to this article, the objective of this part 1 is to ingest data from the New York Times API, run HTML scraping, personality detection and expose it to a Kafka topic, as depicted by the architecture below: The end goal is thus to implement a NiFi flow like the following: To implement it, this article will be divided in 4 section: Section 1: Create a NYT API consumer Section 2: Create a custom BoilerPipe Extractor Processor Section 3: Create a custom Mairesse Personality Processor Section 4: Create and configure a Kafka topic Section 1: Create a NYT API consumer Disclaimer: I basically copy-pasted this from my previous article, Determining the big 5 traits of Personality Psychology of news articles using NiFi, Hive & Zeppelin. Step 1: Obtaining an API Key This step is very straight forward. Go to https://developer.nytimes.com/signup and sign-up for a key. Step 2: Configuring InvokeHTTP The InvokeHTTP is

IntroductionA few weeks ago, I published an article called Determining the big 5 traits of Personality Psychology of news articles using NiFi, Hive & Zeppelin. Since then, I have worked diligently to improve on this first iteration, with the objective to mock up at the heart of every company today: create an end-to-end platform that uses machine learning to not only generate insights, but keeps improving and feeding consumer applications.While doing this work was a way for me to get into the needy-greedy of the latest Hortonworks tools, I decided to share it with the world in the form of a series of tutorial articles, because I believe it is a great way to get familiar with the stack. Architecture overviewLuckily, the Hortonworks platform has all the elements needed to create this end-to-end platform. The figure below gives an overview of this series of articles architecture:As you can see, the goal of this platform is to: Ingest data from news articles (directly from the NYT API a

Env : HDP 2.52 node kafka cluster having topic name 'testtopic' with partition set as 2 and replication set as 2.I am running two consumer with consumer id 'test'.1. what happen when Consumer start freshConsumer NetworkClient will request metadata <- return cluster information2016-12-17 23:21:05 DEBUG clients.NetworkClient:619 - Sending metadata request ClientRequest(expectResponse=true, callback=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=1,client_id=consumer-1}, body={topics=[testtopic]}), isInitiatedByNetworkClient, createdTimeMs=1481997065894, sendTimeMs=0) to node -2 2016-12-17 23:21:06 DEBUG clients.Metadata:172 - Updated cluster metadata version 2 to Cluster(nodes = [Node(1002, rkk2.hdp.local, 6667), Node(1001, rkk1.hdp.local, 6667)], partitions = [Partition(topic = testtopic, partition = 1, leader = 1002, replicas = [1002,1001,], isr = [1002,1001,], Partition(topic = testtopic, partition = 0, leader = 1001, replicas = [1002,1001,], isr = [1001,10

Ambari 2.7 provides UI for API to learn and perform many of the operations. You can launch the UI and play - http://AMBARI-UI:8080/api-docs/For example: You can delete the Host by simply entering the hostname in the UI.1. Go to http://AMBARI-HOST:8080/api-docs/#!/Hosts/deleteHost2. Move to "Delete a host" section3. Enter the "hostname" 4. click on "Try" .Window popup shows the API response with response (errors) message and code like below.Similar many other Services/Groups/users API can be tried in the same page. This feature makes life easies to find and use the APIs.

Starting from Ambari 2.7 there are 2 new features to track who did what.1. In operations history - for every operation there is a user name associated. 2. In Prior versions there is no way to find out who launched Install/Add wizards. Starting from 2.7 there is Username name being shows in the UI.These features make Administrators life easier to track who did what.

With Ambari 2.6.0 or 2.5.x version if you try to use 2.6.3 ( http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-91.xml) or 2.6.4 -) http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.3.0/HDP-2.6.3.0-235.xml ) VDF files, it fails to load/parse the files - would see below error in the logs.f"like An internal system exception occurred: Could not load url from http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-91.xml. null"Those VDF files has GPL tag URLs and these ambari versions can't understand it - hence fails with the above error. You can consider Upgrading the Ambari to 2.6.1 or later versions. If you want to continue using 2.6.0 or earloer Ambari versions then you may have to create private repo and download GPL binaries and work without GPL tags in VDF file. you can contact Hortonworks support for any further help.

Mainframe Offload is becoming the talk of most of the organizations , as they are trying to reduce the cost and move into the latest next generation technologies. Organizations are running critical business applications on mainframe systems that are generating and processing huge volume of data which are tagged with pretty high maintenance cost. With Big Data becoming more famous and every industry is trying to leverage the capability of the Open Source Technologies, organizations are now trying move some or all of their applications to the Open Source. Since the open source systems platforms like Hadoop ecosystems have become more robust , flexible , cheaper , better performing than the traditional systems, the current trend in offloading the legacy systems is becoming more popular. Keeping this in mind , in this article we will discuss about one of the topics, how to offload the data from the the legacy systems like Mainframes into the next generation technologies like Hadoop.The suc

Hortonworks had released HDP 2.6 in Q2 2017. Isilon engineering team worked closely with Hortonworks engineering team to validate HDP 2.6 with OneFS. One of the features of HDP 2.6 was LLAP (Long Live And Process). LLAP enables as fast as sub-second SQL analytics on Hadoop by intelligently caching data in memory with persistent servers that instantly process SQL queries.However, please note that LLAP is officially NOT SUPPORTED with OneFS. Customers are advised NOT to use LLAP (even though it may seem to work after config changes) because it could potentially result in data inconsistency issueAdditional DetailsBy default, LLAP uses “inode paths” to access data from file system. OneFS currently does not support referencing files by inode. So queries coming through LLAP will fail on OneFS. However, it is possible to configure Hive LLAP to NOT use inode path by setting hive.llap.io.use.fileid.path = falseWith this configuration change, the queries will be executed and this will make LLAP

After an unsuccessful upgrade, I was forced to completely remove HDP 2.4, Ambari 2.5 and install HDP 2.6. I wanted to avoid reinstalling the OS, so I took advantage of this instruction. Unfortunately, it is not complete. For the problem-free installation of HDP 2.6, you also need to do things like removing service users, cleaning the cron.So here is my plan of action:1. Stop all services in Ambari or kill them. In my case, Ambari damaged his database during downgrade and could not start. So I manually killed all the processes on all nodes:ps –u hdfs (see list of all services below)kill PID2. Run python script on all cluster nodespython /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users3. Remove Hadoop packages on all nodesyum remove hive\*yum remove oozie\*yum remove pig\*yum remove zookeeper\*yum remove tez\*yum remove hbase\*yum remove ranger\*yum remove knox\*yum remove storm\*yum remove accumulo\*yum remove falcon\*yum remove ambari-metrics-hadoop-si

Sqoop has been popular to be a client interfacing between relational databases and Hive / HDFS etc for importing and exporting datasets. It also supports importing table to HBase as well. The most common use case is to import data in bulk from Databases such as Oracle for the initial loads and then using tools like GoldenGate to continuously replicate data. In this article, we will import 4 million rows from a Mysql table to HBase table. Steps are as follows:Create a table in mysqlCREATE TABLE sqooptest (id int NOT NULL PRIMARY KEY AUTO_INCREMENT, val int);Create a procedure in Mysql to generate some random data in the columns defined.DELIMITER $$CREATE PROCEDURE prepare_data1()BEGINDECLARE i INT DEFAULT 100;WHILE i < 4000000 DOINSERT INTO sqooptest (val) VALUES (i);SET i = i + 1;END WHILE;END$$DELIMITER ;When you call this procedure , you'll have 4 million records in mysql table :CALL prepare_data1();Lets Create HBase table from HBase shell.Shell > create ‘sqooptest’, ‘A’insert

EnvironmentCentOS 7Ambari 2.7.0.0Error:After ambari-server setup, and deploying the desired configuration via Ambari, found this error in the ambari-server.log. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Courier New'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)}span.s1 {font-variant-ligatures: no-common-ligatures}span.s2 {font-variant-ligatures: no-common-ligatures; color: #000000; background-color: rgba(40, 254, 20, 0.9)}2018-08-21 20:09:05,733 WARN [ambari-client-thread-36] HttpChannel:507 - /api/v1/clusters//requests/3org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL was not normalized. at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:123) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:193) at org.springframework.security.web.FilterChainProxy.doFilter(Fil

I recently did a PoC with a customer to integrate NiFi with CDH, part of this was creating external tables in Hive on the newly loaded data. In this article I will share the approaches, useful workarounds, how to customise your own NiFi build for backwards compatibility, and provide a pre-built CDH-compatible Hive Bundle for you to download and try.So first, why is this necessary? Well the short answer is that NiFi 1.x's minimum supported version of Hive is 1.2.x, but CDH uses a fork of Hive1.1.x, which introduces two common backwards compatibility challenges:The first is that it uses an older version of Thrift, so we need to configure NiFi to use this same version if we want to talk directly.The second is that new features introduced after version 1.1.0 aren't available in the CDH release, so we have to stop NiFi from looking for them.The obvious other option here is to work with CDH Hive indirectly, and thus we come to the workarounds.Workarounds: It is very common in PoCs to not ha

IntroductionFor a simple PySpark application, you can use `--py-files` to specify its dependencies. A large PySpark application will have many dependencies, possibly including transitive dependencies. Sometimes a large application needs a Python package that has C code to compile before installation. And, there are times when you might want to run different versions of python for different applications. For such scenarios with large PySpark applications, `--py-files` is inconvenient.Fortunately, in the Python world you can create a virtual environment as an isolated Python runtime environment. We recently enabled virtual environments for PySpark in distributed environments. This eases the transition from local environment to distributed environment with PySpark.In this article, I will talk about how to use virtual environment in PySpark. (This feature is currently only supported in yarn mode.)PrerequisitesHortonworks supports two approaches for setting up a virtual environment: virtual

In addition to the HWX install guides online, this is a great best practices article for groups that want to consider some design options prior to install.http://hortonworks.com/blog/best-practices-in-hdfs-authorization-with-apache-ranger/

Tools in use: HBase shell and Zeppelin User demouser needs access to HBase table called PRICES. User zeppelin needs the same access to run few queries. You can run this demo by using Hortonworks Sandbox

Objective The Hortonworks Sandbox for HDP 2.5 now uses Docker containers, even the VirtualBox version. The process for exposing extra ports on older versions of the sandbox was as simple as setting up additional port forwarding rules in VirtualBox. The new container version of the sandbox requires additional steps; you have to do more than just setup port forwarding rules in VirtualBox. This tutorial will guide you through the process of adding additional ports to the VirtualBox version of the HDP 2.5 sandbox.Prerequisites You should have already downloaded and installed the VirtualBox version of the Hortonworks Sandbox for HDP 2.5 Hortonworks SandboxScope Mac OS X 10.11.6 (El Capitan) VirtualBox 5.1.6 HDP 2.5 VirtualBox SandboxStepsStartup the Sandbox VM Startup your HDP sandbox. It should be called Hortonworks Docker Sandbox within VirtualBox. You should see something similar to this: You can start the sandbox by either double-clicking on the virtual machine or by

Sometimes it is desired to have the logs rotated as well as compressed. We can use log4j extras in order to achieve the same. For processes like NameNode / DataNode...etc we can use the approach described in the article. https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.htmlHowever when we try to use the same approach in Ambari 2.6 for ambari metrics collector log compression and rotation then it will not work and we might see some warnings / errors inside the "" something like following:log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy". log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy". log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'log4j:ERROR No output stream or file set for the appender named [file].(OR)log4j:ERROR A "org.apache.log4j.rolling.SizeBasedTriggeringPolicy" object

ClusterIn Solr, a cluster is a set of Solr nodes operating in coordination with each other via ZooKeeper, and managed as a unit. A cluster may contain many collections. See also SolrCloud.CollectionIn Solr, one or more Documents grouped together in a single logical index using a single configuration and Schema.In SolrCloud a collection may be divided up into multiple logical shards, which may in turn be distributed across many nodes, or in a Single node Solr installation, a collection may be a single Core.CommitTo make document changes permanent in the index. In the case of added documents, they would be searchable after a commit.CoreAn individual Solr instance (represents a logical index). Multiple cores can run on a single node. See also SolrCloud.Key Take Away 1. Solr works on a non master-slave architecture, every solr node is master of its own. Solr nodes uses Zookeper to learn about the state of the cluster. 2. A solr Node (JVM) can host multiple core 3. Core is the place w

Short Description: Example of Play Framework application running integrated Apache Spark applications with direct output to Web. Motivation: Missing articles related to Play and Apache Spark ability to publish output data straight to web. Used components: Play is a high-productivity Java and Scala web application framework that integrates the components and APIs you need for modern web application development. Play is based on a lightweight, stateless, web-friendly architecture and features predictable and minimal resource consumption (CPU, memory, threads) for highly-scalable applications thanks to its reactive model, based on Akka Streams. Link to PlayFramework (https://www.playframework.com) Apache Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop Input Format. It is designed to perform both batch processing

Episode 96 – Roaring newshttps://roaringelephant.org/2018/07/10/episode-96-roaring-news/In this edition of Roaring news, Ward Bekker returns to discuss what is happening in the world of Big Data. Ward brings news on GPUs in supercomputers and how Big Data could be wrong about you. Dave and Jhon found articles on Big data growth visualizations and GDPR. Play in new window | Download (Duration: 46:05 — 31.9MB)10 Charts that will change your perspective of Big Data’s Growth https://www.forbes.com/sites/louiscolumbus/2018/05/23/10-charts-that-will-change-your-perspective-of-big-datas-growth/#1ea595702926New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500 https://www.top500.org/news/new-gpu-accelerated-supercomputers-change-the-balance-of-power-on-the-top500/GDPR: A Call to Remove Technical Debt from Data Science https://medium.com/@kjarmul/gdpr-a-call-to-remove-technical-debt-from-data-science-c103a01c3102Everything big data claims to know about you could b

Episode 95 – DataWorks Summit in San Jose with Ward Bekkerhttps://roaringelephant.org/2018/07/03/episode-95-dataworks-summit-in-san-jose-with-ward-bekker/Since both Dave and Jhon were not able to attend the DataWorks Summit in San Jose a couple of weeks ago, we have a guest, Ward Bekker, who was happy to join and educate us on the subject. Play in new window | Download (Duration: 1:52:50 — 77.7MB)In this episode we discuss the daily keynotes and Wards’ selection of sessions at the Summit ranging from the new things in Yarn 3.0, Materialized views in Hive and much more.Ward Bekker (Linkedin)Pre-Sales Solutions Engineer II @ HortonworksSome of the sessions and topics discussed are:Apache Hadoop State of the union https://dataworkssummit.com/san-jose-2018/session/apache-hadoop-yarn-state-of-the-union-2/What is new in Apache Hive https://dataworkssummit.com/san-jose-2018/session/what-is-new-in-apache-hive/Runing distributed tensorflow in production https://dataworkssummit.com/san-

Launch Cloudbreak on AWSBefore launching Cloudbreak on AWS, review and meet the prerequisites.Meet the PrerequisitesBefore launching Cloudbreak on AWS, you must meet the following prerequisites. AWS AccountAWS RegionSSH Key PairKey Based AuthenticationAWS AccountIn order to launch Cloudbreak on AWS, you must log in to your AWS account. If you don't have an account, you can create one at https://aws.amazon.com/.AWS RegionDecide in which AWS region you would like to launch Cloudbreak. The following AWS regions are supported: SSH Key PairImport an existing Key Pair or generate a new Key Pair in the AWS region which you are planning to use for launching Cloudbreak and clusters. You need this SSH Key Pair to SSH to the Cloudbreak instance and Start Cloudbreak. To do this use the following steps. Navigate to the Amazon EC2 console at https://console.aws.amazon.com/ec2/.Check the region listed in the top right corner to make sure that you are in the correct region.In the left pane, find NETWO