Community Articles

Cloudbreak 2.9.1 maintenance release is now available.If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes.If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template.Useful links:Release notesUpgrade stepsGet started on AWSGet started on AzureGet started on GCP

Below properties can be used to configure the log4j settings for size based compressed backup.Here is an example for Hive service. log4j.appender.DRFA.layout=org.apache.log4j.PatternLayoutlog4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppenderlog4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.DRFA.rollingPolicy.ActiveFileName =${hive.log.dir}/${hive.log.file}.loglog4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%i.log.gzlog4j.appender.DRFA.triggeringPolicy.MaxFileSize=10000log4j.appender.DRFA.rollingPolicy.maxIndex=10For some reason maxfilesize is not working if used like 1MB like that - so use live above.Note: please check console log (.out or .err) for any WARNINGS if does not work

In this article we will be creating a flow to read files from hdfs and insert the same into hive using the putHiveStreaming processor.Before going to NiFi we need update some configurations in Hive.To enable Hive streaming we need to update the following properties hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManagerhive.compactor.initiator.on = truehive.compactor.worker.threads > 0Coming to NiFi we will be making use of the following processors : 1.ListHdfs + FetchHdfs processor – While configuring the List and Fetch HDFS processors we need to make sure that both these processors run on the primary node only so that the flow files are not duplicated across nodes2.Convert Json to Avro processor – PutHiveStreaming processor supports input in the Avro format only. So any Json input needs to be converted to avro format3.PutHiveStreaming processorLets construct the Nifi flow as below : ListHDFS--> FetchHDFS--> ConvertJsonToAvro-->PutHiveStreamingConfiguring the Put

AbstractTensorflow-serving with Apache Hadoop 3.1 and YARN resource management. YARN manages the startup, control and destroys the Tensorflow-serving Docker container in a Hadoop cluster. Client applications using RESTful API calls to communicate with the ML application, i.e. to make predictions. Infrastructure SetupAllocate a Centos 7.4 virtual machine and install:- Docker CE Installation and start Docker- Installation of required packages: amber, curl, wget, unzip, git - Install and Download HDP Sandbox version see this Article Docker on Yarn - Sandboxwget https://github.com/SamHjelmfelt/Amber/archive/master.zipunzip master.zipcd Amber-master./amber.sh createFromPrebuiltSample samples/yarnquickstart/yarnquickstart-sample.iniOutput snippet:...Verifying ambari-agent process status...Ambari Agent successfully startedAgent PID at: /run/ambari-agent/ambari-agent.pidAgent out at: /var/log/ambari-agent/ambari-agent.outAgent log at: /var/log/ambari-agent/ambari-agent.logWaiting for agent he

Sometimes Ambari Metrics stops displaying data in the dashboards.Or You may be getting time out issues that were not solved (see https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se.html)At other times, especially on a new install you may receive this error in the Collector Log2016-01-25 15:33:31,195 WARN org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource: Unable to connect to HBase store using Phoenix.org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG.This is usually due to AMS Data being corrupt. Shut down Ambari Monitors, and Collector via Ambari Cleared out the /var/lib/ambari-metrics-collector dir for fresh restart From Ambari -> Ambari Metrics -> Config -> Advanced ams-hbase-site get the hbase.rootdir and hbase-tmp directory Delete or Move the hbase-tmp and hbase.rootdir directories to an archive folder Started

PurposeUtility helps to collect and compare the hadoop configurations between two or more clusters.Source https://github.com/karthikhw/ambari/blob/master/bin/compare.shUsage ./compare.shInputsHow many clusters you want to compare :Ambari Username :Ambari Password :Hostname :Ambari Server Port :Clustername :Is ambari URL secure (y/n)? (HTTPS/HTTP):Note Run it in ambari serverService configurations are retrieved from ambari via the curl request.

As a Hadoop Admin it’s our responsibility to perform Hadoop Cluster Maintenance frequently. Let’s see what we can do to keep our big elephant happy! 😉1. FileSystem Checks We should check health of HDFS periodically by running fsck command sudo -u hdfs hadoop fsck / This command contacts the Namenode and checks each file recursively which comes under the provided path. Below is the sample output of fsck command sudo -u hdfs hadoop fsck /FSCK started by hdfs (auth:SIMPLE) from /10.0.2.15 for path / at Wed Apr 06 18:47:37 UTC 2016Total size: 1842803118 BTotal dirs: 4612Total files: 11123Total symlinks: 0 (Files currently being written: 4)Total blocks (validated): 11109 (avg. block size 165883 B) (Total open file blocks (not validated): 1)Minimally replicated blocks: 11109 (100.0 %)Over-replicated blocks: 0 (0.0 %)Under-replicated blocks: 11109 (100.0 %)Mis-replicated blocks: 0 (0.0 %)Default replication factor: 3Average block replication: 1.0Corrupt blocks: 0Missing replicas: 22232 (66.6

My blood recently turned from green to blue (after the Hortonworks-Cloudera merger) and I couldn't be more excited to play with new toys. What I am particularly excited about is Cloudera Data Science Workbench. But, like in everything I do, I am very lazy. So here is a quick tutorial to install Altus Director, and use it to deploy a CDH 5.15 + CDSW cluster.Step 1: Install Altus DirectorMany ways to do that, but the one I chose was the AWS install, detailed here: https://www.cloudera.com/documentation/director/latest/topics/director_aws_setup_client.htmlThe installation documentation is very well done, but here are the important excerptsCreate a VPC for your Altus instanceFollow the documentation.Few important points:In the name of laziness I also recommend to add a 0-65535 rule from your personal IP.Your VPC should have an internet gateway associated with it (you could do it without, but would require you manually pulling the CM/CDH software down and make internal repositories within y

IntroductionCFM is out for CDH and that's awesome. While the install documentation is very thorough, I thought I'd share a screenshot based tutorial, avoiding some of the gotchas of the installationPre-RequisitesHealthy CDH 6.2 clusterAdmin privileges on the cluster and its nodeswget installedOpen JDK 1.8 installed (see doc)Zookeeper service installed (see doc)Step 1: Get the CFM CSD FilesConnect to your Cloudera Manager node as root, and enter the following:cd /opt/cloudera/csdwget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFI-1.9.0.1.0.0.0-90.jarwget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFICA-1.9.0.1.0.0.0-90.jarwget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFIREGISTRY-0.3.0.1.0.0.0-90.jarchown cloudera-scm:cloudera-scm NIFI*.jar chmod 644 NIFI*.jar service cloudera-scm-server restartAfter restart, log to CM and restart the Cloudera Management Service:Wait for the restart to finish:Step 2: Get the CFM ParcelGo to the parcels configuration (parcel icon top right):Go to

Customers have asked me about wanting to review ranger audit archive logs stored on HDFS as the UI only shows the Last 90 days of data using Solr infra. I decided to approach the problem using Zeppelin/Spark for a fun example.1. Prerequisites - Zeppelin and Spark2 installed on your system. As well as ranger with ranger audit logs being stored in HDFS. Create a policy in ranger for HDFS to allow your zeppelin user to read and execute recursively for /ranger/audit directory.2. Create your notebook in Zeppelin and create some code like the following example:%spark2.spark// --Specify service and date if you wish//val path = "/ranger/audit/hdfs/20190513/*.log"// --Be brave and map the whole enchiladaval path = "/ranger/audit/*/*/*.log"// --read in the json and drop any malformed jsonval rauditDF = spark.read.option("mode", "DROPMALFORMED").json(path)// --print the schema to review and show me top 20 lines.rauditDF.printSchema()rauditDF.show(20,false)// --Do some spark sql on the data and lo

If Ambari wrongly shows as "Add Service Wizard in Progress", starting from Ambari-2.5.x you can fix this reset the UI state.here is the steps.1. Go to http://AMBARI-HOST:8080/#/experimental 2. Click on "Reset UI state" button3. Now login and check the UI. it will be showing all the service options.Note: This is applicable only from Ambari-2.5.x

This article will demonstrate how to install anaconda on an HDP 3.0.1 instance and the configuration to enable Zeppelin to utilize the anaconda python libraries to use with apache spark.Pre-requisites:bzip2 library needs to be installed prior to installing anacondaStep 1. Download and Install Anacondahttps://www.anaconda.com/download/#linuxFrom the following link, download the anaconda installer. It will be a large bash script around 600MB in size. Please land this file in an appropriate directory like /tmp.Once downloaded, provide the installer with executable permissions.One completed, the installer can be executed. Please execute with a user with sudo privileges. Follow the prompts and accept the license agreements. As a best practice, please install anaconda in a separate directory than the default. /opt is the default for my installations.At the very end of the installation it will prompt the user to initialize the install in the installing users basrc file.If you have a mulit-no

Oozie job submit fails in HDP-3.0 with below error:Error: E0723 : E0723: Unsupported action type, node [hive] type [org.apache.oozie.service.ActionService].It looks like failure is because Oozie is not aware of Hive action type. When I checked value of oozie.service.ActionService.executor.ext.classes in oozie-site.xml, I found that it was set to below value:<property> <name>oozie.service.ActionService.executor.ext.classes</name> <value> org.apache.oozie.action.email.EmailActionExecutor, org.apache.oozie.action.hadoop.ShellActionExecutor, org.apache.oozie.action.hadoop.SqoopActionExecutor, org.apache.oozie.action.hadoop.DistcpActionExecutor</value></property>Note - Hive action type and other supported action types like Spark/Spark2 is missing in this. I'm still working with our engineering to find out why this is happening with recent Oozie version, earlier it used to work without any issues. I will keep you posted on this..To fix this issue:

In the previous articles, we showed how to deploy an HDF 2.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Now we will build on the same cluster and show how to enable kerberos from Ambari using MIT KDC.Summary To achieve this, the high level steps we will follow are:Setup MITC KDCRun Ambari security wizardCreate principal for nifiadmin user in KDCCreate Ranger policy for nifiadmin userDelete certificate from browserLogin to Nifi using KDC principal credentialsSteps 1. Setup MIT KDCHigh level steps to setup KDC:Install KDC rpmsConfigure KDC (krb5.conf)Create KDC databaseStart krb5kdc/kadmin servicesCreate admin principalMake user an administrator by adding to kadm5.aclRestart krb5kdc/kadmin servicesScript to automate KDC setup (run below on Ambari node)export realm=HORTONWORKS.COMexport domain=hortonworks.comexport kdcpassword="BadPass#1"curl -sSL https://gist.github.com/abajwa-hw/f8b83e1c12abb1564531e00836b098fa/raw | sudo -E shTest KDC is up by running below on Amba

Short Description: Here is the steps to use OpenLDAP as Backend for Kerberos Article This is tried for enabling security Kerberos for Cluster and tested with Ambari 2.6.2.0 and HDP 2.6.5 (CENTOS 7) 1-Installation of OpenLDAP yum install openldap-server openldap-clients 2-Kerberos use only LDAPS we have to configure SSL for Openldap (you can use OpenSSL I used tinycert.org to generate my certificates) # mkdir /etc/openldap/cacerts ( Copy the root certificate and the certificate for OpendLdap server and the private keys) # chown ldap:ldap /etc/opendlap/cacerts/* 3- Edit /etc/sysconfig/slapd and set : SLAPD_URLS="ldapi:/// ldap:/// ldaps:///"SLAPD_LDAPS=yes 4- We have to import Root Certificate to Java Keystore of my server keytool -importcert -keystore /usr/jdk64/jdk1.8.0_112/jre/lib/security/cacerts -aliasRootCaLdap -trustcacerts -file cacert.crt 5- Edit /etc/openldap/slapd.d/cn\=config.ldif and set : olcTLSCACertificatePath: /etc/openldap/cacerts/olcTLSCertificateFile: /etc/openlda

Short description:In this article I am going to create a simple Producer to publish messages(tweets) to a kafka topic.Additionally I'm also creating a simple Consumer that subscribes to the kafka topic and reads the messagesCreate the kafka topic:./kafka-topics.sh --create --topic 'kafka-tweets' --partitions 3 --replication-factor 3 --zookeeper <zookeeper node:zk port>Install necessary packages in your python project venv:pip install kafka-python twythonProducer:def main(): # Load credentials from json file with open("twitter_credentials.json", "r") as file: creds = json.load(file) # Instantiate python_tweets = Twython(creds['CONSUMER_KEY'], creds['CONSUMER_SECRET']) # search query query = {'q': 'cloudera', 'result_type': 'mixed', 'count': 100} #result is a python dict of tweets result = python_tweets.search(**query)['statuses'] injest_data(result)To get access to twitter API I need to use my credentials which are stored in "twitter_credentials.j

This article is applicable for anyone deploying a Kafka cluster for production use. It points out certain high level points that you should be thinking about before deploying your cluster. It is divided into 3 parts: Important configurations for the Producer, Broker, Consumer and tips for Performance tuning. Important Client / Producer Configurations: The most important producer configurations: compression The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values arenone, gzip, snappy, or lz4. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression). Name: compression.type Default: None sync vs async production Batching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger batches in a single request. Name: produc

IntroductionControllerStatusReportingTask Logs the 5-minute stats that are shown in the NiFi Summary Page for Processors and Connections. By Default, when configured and started it goes directly to nifi-app.log. These can be configured in the NiFi logging configuration to log to different files, here I try to describe steps to log it to a separate log file with Ambari.Prerequisite1) Assuming you already have HDF-2.x Installed on your VM/Server, Ambari, NiFi is up and running. If not, I would recommend "Ease of Deployment" section of this article to install it [You can also follow this article for Automated installation of HDF cluster or refer hortonworks.com for detailed steps]Configuring "Advanced nifi-node-logback-env" section in Ambari 1. Navigate to your browser window and type in URL for Ambari as below and login to Ambari UI [UI is accessible at port 8080]

Use CaseIn the Internet of Things era, we live in world of connected things like never before with 50 billion assets connected between fleets of vehicles (aircraft, trucks, trains, smart devices) and generating an unprecedented amount of data. Even with the availability of machine-generated data, organizations are still struggling to take advantage of that data to efficiently failure-proof their industrial IOT environment.The business impact across industries of production equipment failure can result in staggering financial costs depending on the industry, but also catastrophic events which can result on human life loss or environmental disaster.When talking to customers about capabilities provided by Hortonworks Data Flow, a recurrent use case being discussed by customers in industries such as oil & gas, utilities, manufacturing and transportation is the ability to take advantage of data in motion to implement predictive maintenance.In this article, I’m going to use the following

When Ambari executes tasks like START/STOP/RESTART/... , it timeouts after waiting 20mins ( 1200 seconds). For example Namenode START command waits for 20mins and then mark the service is Failed/Timeout - in larger clusters NameNodes takes much longer time - How to tell Ambari to wait for much longer?Add below 2 configs and re-start the service - this change increases the timeout value to 1Hour.agent.task.timeout=3600 server.task.timeout=3600Note: This change may not take affect during the Upgrade If it is already scheduled.

Microsoft Azure General Sizing GuidelinesYou need to size and price machine and storage separately.Use Linux VMS on Azure (not to be confused with the Ubuntu Beta offering on HDInsight)If performance is a must, especially with Kafka and Storm, use Premium storage not Standard. Make sure and request Premium Storage (see link below)Do not use A8 machines. Use either A10 or A11’s. A8 is backed by Infiniband which is more expensive and unnecessary for HadoopRecommend D Series and the newer D_v2 Series for Solid State Drives if needed.For Premium Storage use DS_v2 SeriesIt is recommended that Page Blob Storage is used for Hbase as opposed to Block Storage. See link below.Both options will need attached Blob Storage. The 382 GB local disk that comes with the VM is just for temp storage. For Blob Storage, it comes in 1023GB sizes. Each VM has a maximum number of Blob Storage that can be attached. Eg. A10 Vms can have a maximum of 16 * 1TB storage. See the following for more details:

ISSUE:FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) kafka.common.KafkaException: Failed to acquire lock on file .lock in /opt/kafka/log. A Kafka instance in another process or thread is using this directory. at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:98) at kafka.log.LogManager$anonfun$lockLogDirs$1.apply(LogManager.scala:95) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.log.LogManager.lockLogDirs(LogManager.

IntroductionOverviewCloudera/Hortonworks offers today one of the most comprehensive data management platform, with components allowing you data flow management to governed and distributed Data Science workloads.With so many toys to play with, I thought I'd share an easy way to setup a simple cluster that will, using Cloudbreak, setup the following main components on Azure cloud:Hortonworks Data Platform 3.1Hortonworks Data Flow 3.3Data Platform Search 4.0Cloudera Data Science Workbench 1.5Note: This is not a production-ready setup, but merely a first step to customizing your deployment using the Cloudera toolkit.Pre-RequisitesAccount on Azure with permission to assign rolesCloudbreak 2.9 (you can always set one up on your machine using: https://community.hortonworks.com/articles/194076/using-vagrant-and-virtualbox-to-create-a-local-ins.html)Tutorial stepsStep 1: Setup Azure CredentialsStep 2: Setup blueprint and cluster extensionsStep 3: Create clusterStep 1: Setup Azure CredentialsFin

Configuring Hive (streaming)Hive Streaming API allows data to be pumped continuously into Hive. The incoming data can be continuously committed in small batches of records into an existing Hive partition or table. Once data is committed it becomes immediately visible to all Hive queries initiated subsequently. Streaming support is built on top of ACID based insert/update support in HiveStreaming RequirementsThe following settings are required in hive-site.xml to enable ACID support for streaming:hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManagerhive.compactor.initiator.on = true (See more important details here)hive.compactor.worker.threads > 0“stored as orc” must be specified during table creation. Only ORC storage format is supported currently.tblproperties("transactional"="true") must be set on the table during creation.hive.support.concurrency=trueThe Hive table must be bucketed, but not sorted. So something like “clustered by (colName) into 10 buckets” must be sp

Configuring MinifiNow we will install Minifi, which once configured, will be setup on the node capturing data which raspberry pi 3 here. Download the Minifi tar file on node. All the downloads are available on https://nifi.apache.org/minifi/download.html. I have downloaded version 0.51. Download the tar file on the node and unzip.1. wget http://apache.mirrors.hoobly.com/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz 2. tar -xzvf minifi-0.5.0-bin.tar.gz2. Next, we will create the Minifi flow, which will be created on Nifi WebUI. Login to Nifi UI (the one you installed on HDP cluster)3. On NiFi's web UI, place an Input Port "From raspberry" and connect it to a MergeContent processor.4. On the same canvas, create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.5. Inside the process group, create a Remote Process Group (RPG) and give it the URL of y