Resources

In NiFi/HDF, it is possible to create a kind of lookup table of key/value pairs using a DistributedMapCacheServer. The DistributedMapCacheServer is used by various processors such as GetHBase and DetectDuplicate. It can also be leveraged by users in their flows, with the PutDistributedMapCache and FetchDistributedMapCache processors, by specifying a corresponding DistributedMapCacheClientService.Sometimes, however, it might be the case that the user would like to interact with the DistributedMapCacheServer programmatically (and external to NiFi), say for removing specific entries, inserting/populating entries, etc. To that end I have written a Groovy script (dcachegroovy.txt, rename to dcache.groovy) to allow manipulation of the entries in a DistributedMapCacheServer from the command-line, e.g.The usage is as follows:Usage: groovy dcache.groovy <hostname> <port> <command> <args> Where <command> is one of the following: get: Retrieves the values for the key

IntroductionTime for the tutorial 1 of a series detailing how to go from AI to Edge!Note: all code/files referenced in this tutorial can be found on my github, here.AgendaThis tutorial is divided in the following sections:Section 1: Create a custom Docker container running Jupyter for CDSWSection 2: Automate Jupyter launch in a CDSW projectSection 3: Train and save a model reading MNSIT databaseSection 1: Create a custom Docker container running Jupyter for CDSWThis is fairly straight forward to implement, as it is detailed in the official documentation.Note: make sure that dock is signed in with your Dockerhub username/password (not email) otherwise the docker push will not work.Step 1: Create a repository in docker hubGo to docker hub and sign in with your account. Create a new repository as follows:You should see something like this:Step 2: Creating a custom docker fileGo to a folder on your computer can create this docker file (saving it as Dockerfile😞FROM docker.repository.cloude

IntroductionMachine Learning and Artificial Intelligence frameworks are numerous and their impact on the future of computer science need no emphasis. However, deploying these models can be complex and fairly manual without the appropriate multi-function ecosystem, especially when deploying these models to the edge.Luckily, Cloudera comprehensive data management suite make this endeavor very easy.During this series, I will present how to create a Deep Learning model trained to read digits from the MNIST database and deploy it to the edge.This article is an introduction to the architecture and pre-requisites necessary for this tutorial. It will refer to sub articles that will be tutorials that anyone can follow to implement understand how to take an AI model and operationalize it to the edgeArchitecture overviewThe figure below gives a highlight of my hybrid cloud platform:As you can see, it is comprised of the three main functions:Cloudera Data Science Workbench: data science hub used t

Introduction Parquet is a famous file format used with several tools such as Spark. NiFi can be used to easily convert data from different formats such as Avro, CSV or JSON to Parquet. This article explains how to convert data from JSON to Parquet using the PutParquet processor. Implementation Define a schema for the source data In this article I'll be using JSON data source with the following structure: {"created_at" : "Tue Oct 10 21:47:12 CEST 2017","id_store" : 4,"event_type" : "store capacity","id_transaction" : "6594277248900858122","id_product" : 319,"value_product" : 1507664832649} Since we will be using a record based processor, we need to define a schema for our data. This will be an Avro schema but it can be used with other types as well. It's only "a common langage" that helps us describe a schema. The Avro Schema for my data is the following: { "namespace": "nifi", "name": "store_event", "type": "record", "fields": [ { "name": "created_at", "type": "string" }, { "name": "

Hello,My client wants to delegate to certain users (of his company and external) the opening of support cases, how can it be configured?Thanks in advance, Iñigo Perez

1. Goal Security is one of fundamental features for enterprise adoption. Specifically, for SQL users, row/column-level access control is important. However, when a cluster is used as a data warehouse accessed by various user groups via different ways like Apache Spark™ 1.6/2.1 and Apache Hive, it is difficult to guarantee access control in a consistent way. In this article, we show the use of Apache Ranger™ to manage access control policy for Spark SQL. This enhancement is done by integrating Apache Ranger and Apache Spark via `spark-llap`. We also show how to provide finer grained access control(row/column level filtering and column masking) to Apache Spark.2. Key Features Shared Policies: The data in a cluster can be shared securely and consistently controlled by the shared access rules between Apache Spark and Apache Hive. Audits: All security activities can be monitored and searched in a single place, i.e., Apache Ranger. Resources management: Each user can use different queues w

Objective This tutorial walks you through a NiFI flow that utilizes the ConvertRecord processor and Record Reader/Writer controller services to easily convert a CVS file into JSON format. Additionally, the flow is modified to also convert the CSV file to Avro and XML formats. Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0Convert CSV to JSONSupport Files Here is a template of the flow discussed in this tutorial: convert-csv-to-json.xml Here is the CSV file used in the flow: users.txt (Change the extension from .txt to .csv after downloading) Note: The CSV data originated from the site, RandomUser. This useful site provides a free API to pull down randomly generated user data. For example: https://randomuser.me/api/0.6/?results=10&forma

Cloudbreak 2.9.1 maintenance release is now available.If you are using an earlier version of Cloudbreak, you can upgrade now to pick up the latest bug fixes.If you are new to Cloudbreak, you can get started by launching Cloudbreak on AWS, Azure, GCP, or OpenStack from a template.Useful links:Release notesUpgrade stepsGet started on AWSGet started on AzureGet started on GCP

Below properties can be used to configure the log4j settings for size based compressed backup.Here is an example for Hive service. log4j.appender.DRFA.layout=org.apache.log4j.PatternLayoutlog4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppenderlog4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy log4j.appender.DRFA.rollingPolicy.ActiveFileName =${hive.log.dir}/${hive.log.file}.loglog4j.appender.DRFA.rollingPolicy.FileNamePattern=${hive.log.dir}/${hive.log.file}-.%i.log.gzlog4j.appender.DRFA.triggeringPolicy.MaxFileSize=10000log4j.appender.DRFA.rollingPolicy.maxIndex=10For some reason maxfilesize is not working if used like 1MB like that - so use live above.Note: please check console log (.out or .err) for any WARNINGS if does not work

In this article we will be creating a flow to read files from hdfs and insert the same into hive using the putHiveStreaming processor.Before going to NiFi we need update some configurations in Hive.To enable Hive streaming we need to update the following properties hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManagerhive.compactor.initiator.on = truehive.compactor.worker.threads > 0Coming to NiFi we will be making use of the following processors : 1.ListHdfs + FetchHdfs processor – While configuring the List and Fetch HDFS processors we need to make sure that both these processors run on the primary node only so that the flow files are not duplicated across nodes2.Convert Json to Avro processor – PutHiveStreaming processor supports input in the Avro format only. So any Json input needs to be converted to avro format3.PutHiveStreaming processorLets construct the Nifi flow as below : ListHDFS--> FetchHDFS--> ConvertJsonToAvro-->PutHiveStreamingConfiguring the Put

AbstractTensorflow-serving with Apache Hadoop 3.1 and YARN resource management. YARN manages the startup, control and destroys the Tensorflow-serving Docker container in a Hadoop cluster. Client applications using RESTful API calls to communicate with the ML application, i.e. to make predictions. Infrastructure SetupAllocate a Centos 7.4 virtual machine and install:- Docker CE Installation and start Docker- Installation of required packages: amber, curl, wget, unzip, git - Install and Download HDP Sandbox version see this Article Docker on Yarn - Sandboxwget https://github.com/SamHjelmfelt/Amber/archive/master.zipunzip master.zipcd Amber-master./amber.sh createFromPrebuiltSample samples/yarnquickstart/yarnquickstart-sample.iniOutput snippet:...Verifying ambari-agent process status...Ambari Agent successfully startedAgent PID at: /run/ambari-agent/ambari-agent.pidAgent out at: /var/log/ambari-agent/ambari-agent.outAgent log at: /var/log/ambari-agent/ambari-agent.logWaiting for agent he

Sometimes Ambari Metrics stops displaying data in the dashboards.Or You may be getting time out issues that were not solved (see https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se.html)At other times, especially on a new install you may receive this error in the Collector Log2016-01-25 15:33:31,195 WARN org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource: Unable to connect to HBase store using Phoenix.org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG.This is usually due to AMS Data being corrupt. Shut down Ambari Monitors, and Collector via Ambari Cleared out the /var/lib/ambari-metrics-collector dir for fresh restart From Ambari -> Ambari Metrics -> Config -> Advanced ams-hbase-site get the hbase.rootdir and hbase-tmp directory Delete or Move the hbase-tmp and hbase.rootdir directories to an archive folder Started

PurposeUtility helps to collect and compare the hadoop configurations between two or more clusters.Source https://github.com/karthikhw/ambari/blob/master/bin/compare.shUsage ./compare.shInputsHow many clusters you want to compare :Ambari Username :Ambari Password :Hostname :Ambari Server Port :Clustername :Is ambari URL secure (y/n)? (HTTPS/HTTP):Note Run it in ambari serverService configurations are retrieved from ambari via the curl request.

As a Hadoop Admin it’s our responsibility to perform Hadoop Cluster Maintenance frequently. Let’s see what we can do to keep our big elephant happy! 😉1. FileSystem Checks We should check health of HDFS periodically by running fsck command sudo -u hdfs hadoop fsck / This command contacts the Namenode and checks each file recursively which comes under the provided path. Below is the sample output of fsck command sudo -u hdfs hadoop fsck /FSCK started by hdfs (auth:SIMPLE) from /10.0.2.15 for path / at Wed Apr 06 18:47:37 UTC 2016Total size: 1842803118 BTotal dirs: 4612Total files: 11123Total symlinks: 0 (Files currently being written: 4)Total blocks (validated): 11109 (avg. block size 165883 B) (Total open file blocks (not validated): 1)Minimally replicated blocks: 11109 (100.0 %)Over-replicated blocks: 0 (0.0 %)Under-replicated blocks: 11109 (100.0 %)Mis-replicated blocks: 0 (0.0 %)Default replication factor: 3Average block replication: 1.0Corrupt blocks: 0Missing replicas: 22232 (66.6

My blood recently turned from green to blue (after the Hortonworks-Cloudera merger) and I couldn't be more excited to play with new toys. What I am particularly excited about is Cloudera Data Science Workbench. But, like in everything I do, I am very lazy. So here is a quick tutorial to install Altus Director, and use it to deploy a CDH 5.15 + CDSW cluster.Step 1: Install Altus DirectorMany ways to do that, but the one I chose was the AWS install, detailed here: https://www.cloudera.com/documentation/director/latest/topics/director_aws_setup_client.htmlThe installation documentation is very well done, but here are the important excerptsCreate a VPC for your Altus instanceFollow the documentation.Few important points:In the name of laziness I also recommend to add a 0-65535 rule from your personal IP.Your VPC should have an internet gateway associated with it (you could do it without, but would require you manually pulling the CM/CDH software down and make internal repositories within y

IntroductionCFM is out for CDH and that's awesome. While the install documentation is very thorough, I thought I'd share a screenshot based tutorial, avoiding some of the gotchas of the installationPre-RequisitesHealthy CDH 6.2 clusterAdmin privileges on the cluster and its nodeswget installedOpen JDK 1.8 installed (see doc)Zookeeper service installed (see doc)Step 1: Get the CFM CSD FilesConnect to your Cloudera Manager node as root, and enter the following:cd /opt/cloudera/csdwget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFI-1.9.0.1.0.0.0-90.jarwget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFICA-1.9.0.1.0.0.0-90.jarwget http://archive.cloudera.com/CFM/csd/1.0.0.0/NIFIREGISTRY-0.3.0.1.0.0.0-90.jarchown cloudera-scm:cloudera-scm NIFI*.jar chmod 644 NIFI*.jar service cloudera-scm-server restartAfter restart, log to CM and restart the Cloudera Management Service:Wait for the restart to finish:Step 2: Get the CFM ParcelGo to the parcels configuration (parcel icon top right):Go to

Customers have asked me about wanting to review ranger audit archive logs stored on HDFS as the UI only shows the Last 90 days of data using Solr infra. I decided to approach the problem using Zeppelin/Spark for a fun example.1. Prerequisites - Zeppelin and Spark2 installed on your system. As well as ranger with ranger audit logs being stored in HDFS. Create a policy in ranger for HDFS to allow your zeppelin user to read and execute recursively for /ranger/audit directory.2. Create your notebook in Zeppelin and create some code like the following example:%spark2.spark// --Specify service and date if you wish//val path = "/ranger/audit/hdfs/20190513/*.log"// --Be brave and map the whole enchiladaval path = "/ranger/audit/*/*/*.log"// --read in the json and drop any malformed jsonval rauditDF = spark.read.option("mode", "DROPMALFORMED").json(path)// --print the schema to review and show me top 20 lines.rauditDF.printSchema()rauditDF.show(20,false)// --Do some spark sql on the data and lo

Hello All and good morning. When I was registering myself and my company, i didn't add a company description because I was thinking it will be possible to do it later. I dont find now any way to do it. How could I resolve this issue? Stefano

If Ambari wrongly shows as "Add Service Wizard in Progress", starting from Ambari-2.5.x you can fix this reset the UI state.here is the steps.1. Go to http://AMBARI-HOST:8080/#/experimental 2. Click on "Reset UI state" button3. Now login and check the UI. it will be showing all the service options.Note: This is applicable only from Ambari-2.5.x

This article will demonstrate how to install anaconda on an HDP 3.0.1 instance and the configuration to enable Zeppelin to utilize the anaconda python libraries to use with apache spark.Pre-requisites:bzip2 library needs to be installed prior to installing anacondaStep 1. Download and Install Anacondahttps://www.anaconda.com/download/#linuxFrom the following link, download the anaconda installer. It will be a large bash script around 600MB in size. Please land this file in an appropriate directory like /tmp.Once downloaded, provide the installer with executable permissions.One completed, the installer can be executed. Please execute with a user with sudo privileges. Follow the prompts and accept the license agreements. As a best practice, please install anaconda in a separate directory than the default. /opt is the default for my installations.At the very end of the installation it will prompt the user to initialize the install in the installing users basrc file.If you have a mulit-no

Oozie job submit fails in HDP-3.0 with below error:Error: E0723 : E0723: Unsupported action type, node [hive] type [org.apache.oozie.service.ActionService].It looks like failure is because Oozie is not aware of Hive action type. When I checked value of oozie.service.ActionService.executor.ext.classes in oozie-site.xml, I found that it was set to below value:<property> <name>oozie.service.ActionService.executor.ext.classes</name> <value> org.apache.oozie.action.email.EmailActionExecutor, org.apache.oozie.action.hadoop.ShellActionExecutor, org.apache.oozie.action.hadoop.SqoopActionExecutor, org.apache.oozie.action.hadoop.DistcpActionExecutor</value></property>Note - Hive action type and other supported action types like Spark/Spark2 is missing in this. I'm still working with our engineering to find out why this is happening with recent Oozie version, earlier it used to work without any issues. I will keep you posted on this..To fix this issue:

In the previous articles, we showed how to deploy an HDF 2.0 cluster, enable SSL for Nifi and setup the Ranger Nifi plugin. Now we will build on the same cluster and show how to enable kerberos from Ambari using MIT KDC.Summary To achieve this, the high level steps we will follow are:Setup MITC KDCRun Ambari security wizardCreate principal for nifiadmin user in KDCCreate Ranger policy for nifiadmin userDelete certificate from browserLogin to Nifi using KDC principal credentialsSteps 1. Setup MIT KDCHigh level steps to setup KDC:Install KDC rpmsConfigure KDC (krb5.conf)Create KDC databaseStart krb5kdc/kadmin servicesCreate admin principalMake user an administrator by adding to kadm5.aclRestart krb5kdc/kadmin servicesScript to automate KDC setup (run below on Ambari node)export realm=HORTONWORKS.COMexport domain=hortonworks.comexport kdcpassword="BadPass#1"curl -sSL https://gist.github.com/abajwa-hw/f8b83e1c12abb1564531e00836b098fa/raw | sudo -E shTest KDC is up by running below on Amba

Short Description: Here is the steps to use OpenLDAP as Backend for Kerberos Article This is tried for enabling security Kerberos for Cluster and tested with Ambari 2.6.2.0 and HDP 2.6.5 (CENTOS 7) 1-Installation of OpenLDAP yum install openldap-server openldap-clients 2-Kerberos use only LDAPS we have to configure SSL for Openldap (you can use OpenSSL I used tinycert.org to generate my certificates) # mkdir /etc/openldap/cacerts ( Copy the root certificate and the certificate for OpendLdap server and the private keys) # chown ldap:ldap /etc/opendlap/cacerts/* 3- Edit /etc/sysconfig/slapd and set : SLAPD_URLS="ldapi:/// ldap:/// ldaps:///"SLAPD_LDAPS=yes 4- We have to import Root Certificate to Java Keystore of my server keytool -importcert -keystore /usr/jdk64/jdk1.8.0_112/jre/lib/security/cacerts -aliasRootCaLdap -trustcacerts -file cacert.crt 5- Edit /etc/openldap/slapd.d/cn\=config.ldif and set : olcTLSCACertificatePath: /etc/openldap/cacerts/olcTLSCertificateFile: /etc/openlda

Short description:In this article I am going to create a simple Producer to publish messages(tweets) to a kafka topic.Additionally I'm also creating a simple Consumer that subscribes to the kafka topic and reads the messagesCreate the kafka topic:./kafka-topics.sh --create --topic 'kafka-tweets' --partitions 3 --replication-factor 3 --zookeeper <zookeeper node:zk port>Install necessary packages in your python project venv:pip install kafka-python twythonProducer:def main(): # Load credentials from json file with open("twitter_credentials.json", "r") as file: creds = json.load(file) # Instantiate python_tweets = Twython(creds['CONSUMER_KEY'], creds['CONSUMER_SECRET']) # search query query = {'q': 'cloudera', 'result_type': 'mixed', 'count': 100} #result is a python dict of tweets result = python_tweets.search(**query)['statuses'] injest_data(result)To get access to twitter API I need to use my credentials which are stored in "twitter_credentials.j

This article is applicable for anyone deploying a Kafka cluster for production use. It points out certain high level points that you should be thinking about before deploying your cluster. It is divided into 3 parts: Important configurations for the Producer, Broker, Consumer and tips for Performance tuning. Important Client / Producer Configurations: The most important producer configurations: compression The compression type for all data generated by the producer. The default is none (i.e. no compression). Valid values arenone, gzip, snappy, or lz4. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression). Name: compression.type Default: None sync vs async production Batching is one of the big drivers of efficiency, and to enable batching the Kafka producer will attempt to accumulate data in memory and to send out larger batches in a single request. Name: produc