Resources

IntroductionControllerStatusReportingTask Logs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections. By Default, when configured and started it goes directly to nifi-app.log. These can be configured in the NiFi logging configuration to log to different files, here I try to describe steps to log it to a separate log file with Ambari.Prerequisite1) Assuming you already have HDF-2.x Installed on your VM/Server, Ambari, NiFi is up and running. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps]Configuring "Advanced nifi-node-logback-env" section in Ambari 1. Navigate to your browser window and type in URL for Ambari as below and login to Ambari UI [UI is accessible at port 8080]

Use CaseIn the Internet of Things era, we live in world of connected things like never before with 50 billion assets connected between fleets of vehicles (aircraft, trucks, trains, smart devices) and generating an unprecedented amount of data. Even with the availability of machine-generated data, organizations are still struggling to take advantage of that data to efficiently failure-proof their industrial IOT environment.The business impact across industries of production equipment failure can result in staggering financial costs depending on the industry, but also catastrophic events which can result on human life loss or environmental disaster.When talking to customers about capabilities provided by Hortonworks Data Flow, a recurrent use case being discussed by customers in industries such as oil & gas, utilities, manufacturing and transportation is the ability to take advantage of data in motion to implement predictive maintenance.In this article, I’m going to use the following

When Ambari executes tasks like START/STOP/RESTART/... , it timeouts after waiting 20mins ( 1200 seconds). For example Namenode START command waits for 20mins and then mark the service is Failed/Timeout - in larger clusters NameNodes takes much longer time - How to tell Ambari to wait for much longer?Add below 2 configs and re-start the service - this change increases the timeout value to 1Hour.agent.task.timeout=3600 server.task.timeout=3600Note: This change may not take affect during the Upgrade If it is already scheduled.

Microsoft Azure General Sizing GuidelinesYou need to size and price machine and storage separately.Use Linux VMS on Azure (not to be confused with the Ubuntu Beta offering on HDInsight)If performance is a must, especially with Kafka and Storm, use Premium storage not Standard. Make sure and request Premium Storage (see link below)Do not use A8 machines. Use either A10 or A11’s. A8 is backed by Infiniband which is more expensive and unnecessary for HadoopRecommend D Series and the newer D_v2 Series for Solid State Drives if needed.For Premium Storage use DS_v2 SeriesIt is recommended that Page Blob Storage is used for Hbase as opposed to Block Storage. See link below.Both options will need attached Blob Storage. The 382 GB local disk that comes with the VM is just for temp storage. For Blob Storage, it comes in 1023GB sizes. Each VM has a maximum number of Blob Storage that can be attached. Eg. A10 Vms can have a maximum of 16 * 1TB storage. See the following for more details:

ISSUE:FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) kafka.common.KafkaException: Failed to acquire lock on file .lock in /opt/kafka/log. A Kafka instance in another process or thread is using this directory. at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:98) at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:95) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.log.LogManager.lockLogDirs(LogManager.

IntroductionOverviewCloudera/Hortonworks offers today one of the most comprehensive data management platform, with components allowing you data flow management to governed and distributed Data Science workloads.With so many toys to play with, I thought I'd share an easy way to setup a simple cluster that will, using Cloudbreak, setup the following main components on Azure cloud:Hortonworks Data Platform 3.1Hortonworks Data Flow 3.3Data Platform Search 4.0Cloudera Data Science Workbench 1.5Note: This is not a production-ready setup, but merely a first step to customizing your deployment using the Cloudera toolkit.Pre-RequisitesAccount on Azure with permission to assign rolesCloudbreak 2.9 (you can always set one up on your machine using: https://community.hortonworks.com/articles/194076/using-vagrant-and-virtualbox-to-create-a-local-ins.html)Tutorial stepsStep 1: Setup Azure CredentialsStep 2: Setup blueprint and cluster extensionsStep 3: Create clusterStep 1: Setup Azure CredentialsFin

Configuring Hive (streaming)Hive Streaming API allows data to be pumped continuously into Hive. The incoming data can be continuously committed in small batches of records into an existing Hive partition or table. Once data is committed it becomes immediately visible to all Hive queries initiated subsequently. Streaming support is built on top of ACID based insert/update support in HiveStreaming RequirementsThe following settings are required in hive-site.xml to enable ACID support for streaming:hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManagerhive.compactor.initiator.on = true (See more important details here)hive.compactor.worker.threads > 0“stored as orc” must be specified during table creation. Only ORC storage format is supported currently.tblproperties("transactional"="true") must be set on the table during creation.hive.support.concurrency=trueThe Hive table must be bucketed, but not sorted. So something like “clustered by (colName) into 10 buckets” must be sp

Configuring MinifiNow we will install Minifi, which once configured, will be setup on the node capturing data which raspberry pi 3 here. Download the Minifi tar file on node. All the downloads are available on https://nifi.apache.org/minifi/download.html. I have downloaded version 0.51. Download the tar file on the node and unzip.1. wget http://apache.mirrors.hoobly.com/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz 2. tar -xzvf minifi-0.5.0-bin.tar.gz2. Next, we will create the Minifi flow, which will be created on Nifi WebUI. Login to Nifi UI (the one you installed on HDP cluster)3. On NiFi's web UI, place an Input Port "From raspberry" and connect it to a MergeContent processor.4. On the same canvas, create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.5. Inside the process group, create a Remote Process Group (RPG) and give it the URL of y

Creating NIFI workflowUse below steps to create the whole workflow.1. Open the Nifi Web UI, download the below processors and configure as described.a. MergeContentb. InferAvroSchemac. PutHive3streaming2. Now let’s configure them one by one. Right click each processor and configure as shown below.a. MergeContentb. InferAvroSchema – In the CSV header definition put all the headers of your data separated by delimiter.c. PutHive3Streaming – Here we need to configure the record reader. Right click on the processor > configure > Record reader > create new service.d. Add CSV reader and save.e. Click on the record reader again and then enable by clicking on middle sign on upper right-side corner.f. Connect all three processor and address the errors if any. For any configuration reference, you can try importing the attached workflow.I have attached my workflow in case you want to directly import into your Nifi.Rasp_new.xmlLinks to seriesPart 1, Part 2, Part 3, Part 4, Part 5

NIFI Installation on HDPWe will go through step by step instructions how to setup your own data pipeline. First to save time and resources we will install NiFi on HDP cluster using mpack.1. Install and setup the HDP cluster.2. Install the HDF mpack (versions may vary) on ambari server node and restart the ambari server.1. wget http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.2.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.2.0.0-520.tar.gz2. cp -r /var/lib/ambari-server/resources /var/lib/ambari-server/resources.backup3. ambari-server install-mpack --mpack=hdf-ambari-mpack-3.2.0.0-520.tar.gz  --verbose4. ambari-server restart3. Once we restart the ambari-server, we will be now able to add Nifi service to the same HDP cluster. Two different clusters can be used for HDP and HDF, however we are using only one to reduce the complexity.4. Select NiFi service5. Resume the installation after typing in the Master key, encryption and Nifi CA token password6. Once it is done, valid

INTRODUCTIONFrom Edge to AI: This tutorial is designed to help you walk through the process of creating a work flow to read data from a edge sensors (Raspberry Pi 3 here) and ingest in Hive through NiFi workflow. The MiNiFi flow will push data to a remote NiFi instance which then using Hive streaming features will ingest data to Hive tables. Once ingested, it can be analyzed with hive sql interface, Spark, zeppelin and by numerous other tools designed for Hadoop platform.In the whole process we will be going through the installation and configuration of below working components.NiFi - Apache NiFi commonly called Niagara Files, is an integrated data logistics platform for automating the movement of data between disparate systems. It provides real-time control that makes it easy to manage the movement of data between any source and any destination. It is data source agnostic, supporting disparate and distributed sources of differing formats, schemas, protocols, speeds and sizes such as m

  I need to change my Community Nickname but the option is disabled, is there a way I can change my nickname ??

Steps on how to setup YARN to run docker containers can be found in Part 1: articleIn this article I will show how to run Hive components (Hiverserver2, Metastore) as docker containers in YARN.Metastore will be using a Mysql 8 database also running as a docker container in a local host.Pre-requisites:1. Pull mysql-server image from Docker hub, run image as a docker container, create hive database and permissions for hive user :docker pull mysql/mysql-serverdocker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=admin  --restart=always --name mysqld mysql/mysql-serverdocker exec -it mysqld bashbash-4.2# mysql -u root --password=adminmysql> CREATE DATABASE hive;mysql> CREATE USER 'hive' IDENTIFIED BY 'hive';mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' WITH GRANT OPTION;2. Create user 'hive' and assign to 'hadoop' group:useradd hiveusermod -aG hadoop hiveDockerize Hive:1. Create a yum repo file "hdp.repo" that contains HDP-3.1.0.0 and HDP-UTILS-1.1.0.22 repositories:[HDP-3.

This article talks about two things:1. How to develop a simple python script which returns the result_code like 'CRITICAL', 'WARN', 'OK', 'UNKNOWN' based on certain conditions matching our requirement.2. How to write an alert definition in json format so that we can install those alerts to the Ambari Server in order to get the alerts.The scripts/files attached to this article are "test_alert_disk_space.py" and "alerts.json" that we are going to use as part of the following steps:Step-1). Create a python script as attached "test_alert_disk_space.py" that finds the list of mount points and the it returns the disk usage status of those mount points. The based on the percentages specified it returns the return code as 'CRITICAL', 'WARN', 'OK', 'UNKNOWN'Step-2). Place the file "test_alert_disk_space.py" in the following Path on the ambari-server : "/var/lib/ambari-server/resources/host_scripts/" Example:cp -f test_alert_disk_space.py /var/lib/ambari-server/resources/host_scriptsStep-3).

 How can I change my account email? Thanks

AbstractThis article objective is to cover key design and deployment considerations for High Availability Apache Kafka service.IntroductionKafka’s through its distributed design allows a large number of permanent or ad-hoc consumers, being highly available and resilient to node failures, also supporting automatic recovery. These characteristics make Kafka an ideal fit for communication and integration between components of large scale data systems. Kafka application resilience is not enough to have a true HA system. Consumers and producers or the network design need also to be HA. Zookeepers and brokers need to be able to communicate among themselves, producers and consumers need to be able to access Kafka API. A fast car can reach the maximum speed only on a good track. A swamp is a swamp for a fast or a slow car.Kafka DesignKafka utilizes Zookeeper for storing metadata information about the brokers, topics, and partitions. Writes to Zookeeper are only performed on changes to the memb

I am planning to take the CCP Data Science Essential exam, i dont have access to the forum "CCP: Data Science" (link below). It says i dont have access.  

Joining Collections in SOLR (Part 1) Sometimes you may want to inner join data from one solr connection to another. There is a facility to perform this action using a join query in SOLR. The easiest way to perform the join is by linking a single attribute from one collection to another attribute in another collection. This join works very well for standalone indexes, but does not work well for distributed indexes. To do this in a distributed index, we’ll perform that in part II of this article. To demonstrate, let’s say we have two collections. Sales, which contains the amount of sales by region. And in the other collection called People, which has people categorized by their region and a flag if they are a manager. Let’s say our goal is to find all of the sales by manager. To do this, we will join the collections using region as our join key, and also filter the people data by if they are a manager or not. Here is the filter query (fq) in solr on how to make this happen: fq={!join

This article describes the setup of two separate KDCs in a Master/Slave configuration. This setup will allow two clusters to share a single Kerberos realm, which allows the principals to be recognized between clusters. A use case for this configuration is when a Disaster Recovery cluster is used as a warm standby. The high level information for the article was found at https://web.mit.edu/kerberos/krb5-1.13/doc/admin/install_kdc.html, while the details were worked out through sweat and tears.Execute the following command to install the Master and Slave KDC if the KDC is not already installed: yum install krb5-server The following defines the KDC configuration for both clusters. This file, /etc/krb5.conf, must be copied to each node in the cluster.[libdefaults] renew_lifetime = 7d forwardable = true default_realm = CUSTOMER.HDP ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false udp_preference_limit=1[domain_realm] customer.com = CUSTOMER.HDP .customer.com = C

  Short Description: This article targets to describe and demonstrate Apache Hive Warehouse Connector which is a newer generation to read and write data between Apache Spark and Apache Hive. 1. Motivation Apache Spark and Apache Hive integration has always been an important use case and continues to be so. Both provide their own efficient ways to process data by the use of SQL, and is used for data stored in distributed file systems. Both provide compatibilities for each other. As both systems evolve, it is critical to find a solution that provides the best of both worlds for data processing needs. In case of Apache Spark, it provides a basic Hive compatibility. It allows an access to tables in Apache Hive and some basic use cases can be achieved by this. However, not all the modern features from Apache Hive are supported, for instance, ACID table in Apache Hive, Ranger integration, Live Long And Process (LLAP), etc. as of Spark 2. Apache Spark supports a pluggable approach for va

HDP Search provides the tools to index data from your HDP cluster to Solr. You can utilize the power of connectors that are shipped with HDPSearch to index data from HDFS, Hive tables, and Spark dataframes to Solr. Once you have your data in Solr, search and querying is simpler. You may find the official Hortonworks documentation for HDP Search 4.0 here: https://docs.hortonworks.com/HDPDocuments/HDPS/HDPS-4.0.0/bk_solr-search-installation/content/ch_hdp-search.html In this article, you may find how you could set up your HDP Cluster with HDP Search using the Solr Management pack shipped by Hortonworks. Additionally, you will find details on how to use connectors for Hive, Spark, and Hdfs to index data to Solr and query from Solr. (This document assumes that you already have an Ambari-2.7.0 + HDP-3.0 cluster up and running) Setup Install Management Pack: Download Solr service mpack on Ambari server node: wget http://public-repo-1.hortonworks.com/HDP-SOLR/hdp-solr-ambari-mp/solr-service

If you would like to upgrade Cloudbreak in an environment with no internet access, refer to the following documentation:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/upgrade/content/cb_upgrade-cloudbreak-no-internet.html

If you would like to configure Cloudbreak to use an external database that uses SSL, refer to the following documentation:https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.9.0/configure/content/cb_configure-external-ssl.html

Due to AMBARI-24283 BUG some times there would be warnings like below,2018-08-09 09:38:27,666 WARN - You have config groups present in the database with no corresponding service found, [(ConfigGroup, Service) => ( HDFS_DL111, null ), ( YARN_2222, null )]. Run --auto-fix-database to fix this automatically. Possible root cause: In some of the Ambari 2.5.x/2.6.x versions, service DELETE does not delete config groups cleanly and it shows the WARNING while starting ambari-server.To fix above WARNINGS below steps can be followed.1. Take Ambari database backup.2. Run below queriesdelete from confgroupclusterconfigmapping where config_group_id in (select group_id from configgroup where group_name in ( 'HDFS_DL111', 'YARN_2222'));delete from configgrouphostmapping where config_group_id in (select group_id from configgroup where group_name in ( 'HDFS_DL111', 'YARN_2222'));delete from configgroup where group_name in ( 'HDFS_DL111', 'YARN_2222');

ISSUE: Sqooping data from hdfs to mysql db works when we run it from command line.sqoop import --connect jdbc:mysql://mysqlserver.somedomain.com:3306/sample --username user1 --password password --table sample_test --target-dir /user/user1/data When we attempt to run the same in an oozie workflow the job fails with the following error;Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] SOLUTION: Setting the path to the oozie.libpath in the job.properties and copying the database connector jars to this directory resolved this issue. Original job.properties:oozie.use.system.libpath=Truesecurity_enabled=TruenameNode=hdfs://HD0credentials={u'hcat': {'xml_name': u'hcat', 'properties': [('hcat.metastore.uri', u'thrift://server.somedomain.com:9083'), ('hcat.metastore.principal', u'hive/server.somedomain.com@SOMEREALM.COM')]}, u'hive2': {'xml_name': u'hive2', 'properties': [('hive2.jdbc.url', 'jdbc:hive2://server.somedomain.com:10000/default'), ('hive