Resources

Ambari 2.7 provides UI for API to learn and perform many of the operations. You can launch the UI and play - http://AMBARI-UI:8080/api-docs/For example: You can delete the Host by simply entering the hostname in the UI.1. Go to http://AMBARI-HOST:8080/api-docs/#!/Hosts/deleteHost2. Move to "Delete a host" section3. Enter the "hostname" 4. click on "Try" .Window popup shows the API response with response (errors) message and code like below.Similar many other Services/Groups/users API can be tried in the same page. This feature makes life easies to find and use the APIs.

Starting from Ambari 2.7 there are 2 new features to track who did what.1. In operations history - for every operation there is a user name associated. 2. In Prior versions there is no way to find out who launched Install/Add wizards. Starting from 2.7 there is Username name being shows in the UI.These features make Administrators life easier to track who did what.

With Ambari 2.6.0 or 2.5.x version if you try to use 2.6.3 ( http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-91.xml) or 2.6.4 -) http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.3.0/HDP-2.6.3.0-235.xml ) VDF files, it fails to load/parse the files - would see below error in the logs.f"like An internal system exception occurred: Could not load url from http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-91.xml. null"Those VDF files has GPL tag URLs and these ambari versions can't understand it - hence fails with the above error. You can consider Upgrading the Ambari to 2.6.1 or later versions. If you want to continue using 2.6.0 or earloer Ambari versions then you may have to create private repo and download GPL binaries and work without GPL tags in VDF file. you can contact Hortonworks support for any further help.

Mainframe Offload is becoming the talk of most of the organizations , as they are trying to reduce the cost and move into the latest next generation technologies. Organizations are running critical business applications on mainframe systems that are generating and processing huge volume of data which are tagged with pretty high maintenance cost. With Big Data becoming more famous and every industry is trying to leverage the capability of the Open Source Technologies, organizations are now trying move some or all of their applications to the Open Source. Since the open source systems platforms like Hadoop ecosystems have become more robust , flexible , cheaper , better performing than the traditional systems, the current trend in offloading the legacy systems is becoming more popular. Keeping this in mind , in this article we will discuss about one of the topics, how to offload the data from the the legacy systems like Mainframes into the next generation technologies like Hadoop.The suc

Hortonworks had released HDP 2.6 in Q2 2017. Isilon engineering team worked closely with Hortonworks engineering team to validate HDP 2.6 with OneFS. One of the features of HDP 2.6 was LLAP (Long Live And Process). LLAP enables as fast as sub-second SQL analytics on Hadoop by intelligently caching data in memory with persistent servers that instantly process SQL queries.However, please note that LLAP is officially NOT SUPPORTED with OneFS. Customers are advised NOT to use LLAP (even though it may seem to work after config changes) because it could potentially result in data inconsistency issueAdditional DetailsBy default, LLAP uses “inode paths” to access data from file system. OneFS currently does not support referencing files by inode. So queries coming through LLAP will fail on OneFS. However, it is possible to configure Hive LLAP to NOT use inode path by setting hive.llap.io.use.fileid.path = falseWith this configuration change, the queries will be executed and this will make LLAP

After an unsuccessful upgrade, I was forced to completely remove HDP 2.4, Ambari 2.5 and install HDP 2.6. I wanted to avoid reinstalling the OS, so I took advantage of this instruction. Unfortunately, it is not complete. For the problem-free installation of HDP 2.6, you also need to do things like removing service users, cleaning the cron.So here is my plan of action:1. Stop all services in Ambari or kill them. In my case, Ambari damaged his database during downgrade and could not start. So I manually killed all the processes on all nodes:ps –u hdfs (see list of all services below)kill PID2. Run python script on all cluster nodespython /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users3. Remove Hadoop packages on all nodesyum remove hive\*yum remove oozie\*yum remove pig\*yum remove zookeeper\*yum remove tez\*yum remove hbase\*yum remove ranger\*yum remove knox\*yum remove storm\*yum remove accumulo\*yum remove falcon\*yum remove ambari-metrics-hadoop-si

Sqoop has been popular to be a client interfacing between relational databases and Hive / HDFS etc for importing and exporting datasets. It also supports importing table to HBase as well. The most common use case is to import data in bulk from Databases such as Oracle for the initial loads and then using tools like GoldenGate to continuously replicate data. In this article, we will import 4 million rows from a Mysql table to HBase table. Steps are as follows:Create a table in mysqlCREATE TABLE sqooptest (id int NOT NULL PRIMARY KEY AUTO_INCREMENT, val int);Create a procedure in Mysql to generate some random data in the columns defined.DELIMITER $$CREATE PROCEDURE prepare_data1()BEGINDECLARE i INT DEFAULT 100;WHILE i < 4000000 DOINSERT INTO sqooptest (val) VALUES (i);SET i = i + 1;END WHILE;END$$DELIMITER ;When you call this procedure , you'll have 4 million records in mysql table :CALL prepare_data1();Lets Create HBase table from HBase shell.Shell > create ‘sqooptest’, ‘A’insert

EnvironmentCentOS 7Ambari 2.7.0.0Error:After ambari-server setup, and deploying the desired configuration via Ambari, found this error in the ambari-server.log. p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px 'Courier New'; color: #28fe14; background-color: #000000; background-color: rgba(0, 0, 0, 0.9)}span.s1 {font-variant-ligatures: no-common-ligatures}span.s2 {font-variant-ligatures: no-common-ligatures; color: #000000; background-color: rgba(40, 254, 20, 0.9)}2018-08-21 20:09:05,733 WARN [ambari-client-thread-36] HttpChannel:507 - /api/v1/clusters//requests/3org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL was not normalized. at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:123) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:193) at org.springframework.security.web.FilterChainProxy.doFilter(Fil

I recently did a PoC with a customer to integrate NiFi with CDH, part of this was creating external tables in Hive on the newly loaded data. In this article I will share the approaches, useful workarounds, how to customise your own NiFi build for backwards compatibility, and provide a pre-built CDH-compatible Hive Bundle for you to download and try.So first, why is this necessary? Well the short answer is that NiFi 1.x's minimum supported version of Hive is 1.2.x, but CDH uses a fork of Hive1.1.x, which introduces two common backwards compatibility challenges:The first is that it uses an older version of Thrift, so we need to configure NiFi to use this same version if we want to talk directly.The second is that new features introduced after version 1.1.0 aren't available in the CDH release, so we have to stop NiFi from looking for them.The obvious other option here is to work with CDH Hive indirectly, and thus we come to the workarounds.Workarounds: It is very common in PoCs to not ha

IntroductionFor a simple PySpark application, you can use `--py-files` to specify its dependencies. A large PySpark application will have many dependencies, possibly including transitive dependencies. Sometimes a large application needs a Python package that has C code to compile before installation. And, there are times when you might want to run different versions of python for different applications. For such scenarios with large PySpark applications, `--py-files` is inconvenient.Fortunately, in the Python world you can create a virtual environment as an isolated Python runtime environment. We recently enabled virtual environments for PySpark in distributed environments. This eases the transition from local environment to distributed environment with PySpark.In this article, I will talk about how to use virtual environment in PySpark. (This feature is currently only supported in yarn mode.)PrerequisitesHortonworks supports two approaches for setting up a virtual environment: virtual

In addition to the HWX install guides online, this is a great best practices article for groups that want to consider some design options prior to install.http://hortonworks.com/blog/best-practices-in-hdfs-authorization-with-apache-ranger/

Tools in use: HBase shell and Zeppelin User demouser needs access to HBase table called PRICES. User zeppelin needs the same access to run few queries. You can run this demo by using Hortonworks Sandbox

Objective The Hortonworks Sandbox for HDP 2.5 now uses Docker containers, even the VirtualBox version. The process for exposing extra ports on older versions of the sandbox was as simple as setting up additional port forwarding rules in VirtualBox. The new container version of the sandbox requires additional steps; you have to do more than just setup port forwarding rules in VirtualBox. This tutorial will guide you through the process of adding additional ports to the VirtualBox version of the HDP 2.5 sandbox.Prerequisites You should have already downloaded and installed the VirtualBox version of the Hortonworks Sandbox for HDP 2.5 Hortonworks SandboxScope Mac OS X 10.11.6 (El Capitan) VirtualBox 5.1.6 HDP 2.5 VirtualBox SandboxStepsStartup the Sandbox VM Startup your HDP sandbox. It should be called Hortonworks Docker Sandbox within VirtualBox. You should see something similar to this: You can start the sandbox by either double-clicking on the virtual machine or by

Sometimes it is desired to have the logs rotated as well as compressed. We can use log4j extras in order to achieve the same. For processes like NameNode / DataNode...etc we can use the approach described in the article. https://community.hortonworks.com/articles/50058/using-log4j-extras-how-to-rotate-as-well-as-zip-th.htmlHowever when we try to use the same approach in Ambari 2.6 for ambari metrics collector log compression and rotation then it will not work and we might see some warnings / errors inside the "" something like following:log4j:WARN Failed to set property [triggeringPolicy] to value "org.apache.log4j.rolling.SizeBasedTriggeringPolicy". log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.FixedWindowRollingPolicy". log4j:WARN Please set a rolling policy for the RollingFileAppender named 'file'log4j:ERROR No output stream or file set for the appender named [file].(OR)log4j:ERROR A "org.apache.log4j.rolling.SizeBasedTriggeringPolicy" object

ClusterIn Solr, a cluster is a set of Solr nodes operating in coordination with each other via ZooKeeper, and managed as a unit. A cluster may contain many collections. See also SolrCloud.CollectionIn Solr, one or more Documents grouped together in a single logical index using a single configuration and Schema.In SolrCloud a collection may be divided up into multiple logical shards, which may in turn be distributed across many nodes, or in a Single node Solr installation, a collection may be a single Core.CommitTo make document changes permanent in the index. In the case of added documents, they would be searchable after a commit.CoreAn individual Solr instance (represents a logical index). Multiple cores can run on a single node. See also SolrCloud.Key Take Away 1. Solr works on a non master-slave architecture, every solr node is master of its own. Solr nodes uses Zookeper to learn about the state of the cluster. 2. A solr Node (JVM) can host multiple core 3. Core is the place w

Short Description: Example of Play Framework application running integrated Apache Spark applications with direct output to Web. Motivation: Missing articles related to Play and Apache Spark ability to publish output data straight to web. Used components: Play is a high-productivity Java and Scala web application framework that integrates the components and APIs you need for modern web application development. Play is based on a lightweight, stateless, web-friendly architecture and features predictable and minimal resource consumption (CPU, memory, threads) for highly-scalable applications thanks to its reactive model, based on Akka Streams. Link to PlayFramework (https://www.playframework.com) Apache Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop Input Format. It is designed to perform both batch processing

Episode 96 – Roaring newshttps://roaringelephant.org/2018/07/10/episode-96-roaring-news/In this edition of Roaring news, Ward Bekker returns to discuss what is happening in the world of Big Data. Ward brings news on GPUs in supercomputers and how Big Data could be wrong about you. Dave and Jhon found articles on Big data growth visualizations and GDPR. Play in new window | Download (Duration: 46:05 — 31.9MB)10 Charts that will change your perspective of Big Data’s Growth https://www.forbes.com/sites/louiscolumbus/2018/05/23/10-charts-that-will-change-your-perspective-of-big-datas-growth/#1ea595702926New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500 https://www.top500.org/news/new-gpu-accelerated-supercomputers-change-the-balance-of-power-on-the-top500/GDPR: A Call to Remove Technical Debt from Data Science https://medium.com/@kjarmul/gdpr-a-call-to-remove-technical-debt-from-data-science-c103a01c3102Everything big data claims to know about you could b

Episode 95 – DataWorks Summit in San Jose with Ward Bekkerhttps://roaringelephant.org/2018/07/03/episode-95-dataworks-summit-in-san-jose-with-ward-bekker/Since both Dave and Jhon were not able to attend the DataWorks Summit in San Jose a couple of weeks ago, we have a guest, Ward Bekker, who was happy to join and educate us on the subject. Play in new window | Download (Duration: 1:52:50 — 77.7MB)In this episode we discuss the daily keynotes and Wards’ selection of sessions at the Summit ranging from the new things in Yarn 3.0, Materialized views in Hive and much more.Ward Bekker (Linkedin)Pre-Sales Solutions Engineer II @ HortonworksSome of the sessions and topics discussed are:Apache Hadoop State of the union https://dataworkssummit.com/san-jose-2018/session/apache-hadoop-yarn-state-of-the-union-2/What is new in Apache Hive https://dataworkssummit.com/san-jose-2018/session/what-is-new-in-apache-hive/Runing distributed tensorflow in production https://dataworkssummit.com/san-

Launch Cloudbreak on AWSBefore launching Cloudbreak on AWS, review and meet the prerequisites.Meet the PrerequisitesBefore launching Cloudbreak on AWS, you must meet the following prerequisites. AWS AccountAWS RegionSSH Key PairKey Based AuthenticationAWS AccountIn order to launch Cloudbreak on AWS, you must log in to your AWS account. If you don't have an account, you can create one at https://aws.amazon.com/.AWS RegionDecide in which AWS region you would like to launch Cloudbreak. The following AWS regions are supported: SSH Key PairImport an existing Key Pair or generate a new Key Pair in the AWS region which you are planning to use for launching Cloudbreak and clusters. You need this SSH Key Pair to SSH to the Cloudbreak instance and Start Cloudbreak. To do this use the following steps. Navigate to the Amazon EC2 console at https://console.aws.amazon.com/ec2/.Check the region listed in the top right corner to make sure that you are in the correct region.In the left pane, find NETWO

SYMPTOM: Installing Ranger Admin with Ambari 2.2.1, I got the following exception during the installation Ranger Admin installation failed - xa_audit_db_postgres.sql DB schema import failed!ERROR: ### 2016-05-10 10:25:47,952 - Execute['python /usr/hdp/current/ranger-admin/db_setup.py'] {'logoutput': True, 'environment': {'RANGER_ADMIN_HOME': '/usr/hdp/current/ranger-admin', 'JAVA_HOME': '/usr/lib/jvm/jre-1.7.0-openjdk.x86_64'}, 'user': 'ranger'} 2016-05-10 10:25:48,081 [I] DB FLAVOR :POSTGRES 2016-05-10 10:25:48,081 [I] --------- Verifying Ranger DB connection --------- 2016-05-10 10:25:48,081 [I] Checking connection 2016-05-10 10:25:48,332 [I] connection success 2016-05-10 10:25:48,332 [I] --------- Verifying Ranger DB tables --------- 2016-05-10 10:25:48,332 [I] Verifying table x_portal_user in database ranger 2016-05-10 10:25:48,583 [I] Table x_portal_user already exists in database ranger 2016-05-10 10:25:48,583 [I] --------- Verifying upgrade history table --------- 2016-05-10 10:

Episode 94 – Roaring newshttps://roaringelephant.org/2018/06/26/episode-94-roaring-news/I this weeks edition of Roaring Big Data News, Dave talks about modernizing Hadoop and a billion java errors. Jhon has an article on improving your learning data sets. We finish with a discussion about the newly released HDP 2.6.5 with an emphasis on the deprecation notices and Yarn Containers. Play in new window | Download (Duration: 37:40 — 26.1MB)Dave Modernizing Hadoop: Reaching the plateau of productivity https://www.zdnet.com/article/modernizing-hadoop-reaching-the-plateau-of-productivity/1 billion Java errors, here’s what causes 97% of them https://blog.takipi.com/we-crunched-1-billion-java-logged-errors-heres-what-causes-97-of-them/https://blog.takipi.com/the-top-10-exceptions-types-in-production-java-applications-based-on-1b-events/Jhon Why you need to improve your training data, and how to do it https://petewarden.com/2018/05/28/why-you-need-to-improve-your-training-data-and-how-t

See Part 1: https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html Augment Your Streaming Ingested Data So with Hortonworks Data Flow : Flow Processing I can augment any stream of data in motion. My GPS data stream from my RPIWZ is returning latitude and longitude. That is valuable data that can be input to a lot of different APIs. One that seemed relevant to me was the current weather conditions where the device was. You could also do some predictive analytics to figure where you might be and check the current weather conditions or a weather forecast depending on how long it will take you to get there. A GPS is giving us latitude and longitude that can be used as input to many kinds of APIs. Once use case that works for a moving device is checking the current weather conditions where you are. Weather Underground has a REST API call that will convert Lat/Long to City/State (there are other APIs that will do this as well,

Short Description: This article includes manual steps to be performed, if a cluster is setup with Hdfs, Ranger and Ranger KMS services and Ranger service is syncing users from LDAP/AD and Hadoop group mapping is configured using SSSD. Steps provided will help Ranger KMS service to resolve groups with spaces or no name as same as in Hadoop.Article:To resolve groups with spaces or no name, Ranger KMS process should have access to hadoop native library path for using JniBasedUnixGroupsMapping class.Steps for Ambari 2.6.x version: Create a sh file ranger-kms-env-javaopts.sh in Ranger KMS conf directory which will specify -Djava.library.path.vim /usr/hdp/current/ranger-kms/conf/ranger-kms-env-javaopts.sh Add below content and save ranger-kms-env-javaopts.sh file.export JAVA_OPTS=" ${JAVA_OPTS} -Djava.library.path=${JAVA_LIBRARY_PATH}:/usr/hdp/current/hadoop-client/lib/native " Update the ownership of ranger-kms-env-javaopts.sh with the user which is used to start Ranger KMS process.

Episode 93 – Apache Kylin: Extreme OLAP Engine for Big Datahttps://roaringelephant.org/2018/06/19/episode-93-apache-kylin-olap-cubes-in-hadoop/In this episode Apache PMC member Dong Li joins us to explains how Apache Kylin can deploy Analytical OLAP cubes in your Big Data environment. http://kylin.apache.org/ Play in new window | Download (Duration: 46:14 — 32.0MB)Dong LiTechnical Partner & Senior Architect of Kyligence (linkedin)PMC Member of Apache Kylinhttp://en.kyligence.io/Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Was trying to dig up some TSP benchmark info for nifi listenhttp, which allows for providing a rest proxy, with no luck, so i tried to create one myself. This is a very rough effort, i can improve it by capturing how many client instances i have running , so we can see the TPS drop and rise as the clients drop and rise. At some point i ran out of servers and resources to run more clients, you will the chart drops off at the end , before going back up. This is where i noticed some of my clients had crashed and i restarted them.Anyway, getting to the matter. PurposeThe benchmark only meassure how much load can a ListenHTTP processor handle , when subjected to real world traffic.SetupThe nifi cluster is setup on an m4.4xlarge ( 16 cores CPU, 32 GB RAM), The node is also hosting the kafka broker and zookeeper. HDF version is 3.1.1The NiFi is a simple Listenhttp processor forwarding to updateattribute. updateattribute burns the flowfile. The idea was to only measure Listenhttp performance