Community Articles

This is a Work In Progress Article. Downloads:HDP/HDF 3 Ambari 2.7 Mpack ELK 6.3.2 with sudo (for non root user install) elasticsearch_mpack-3.0.0.0-1.tar.gzHDP/HDF 3 Ambari 2.7 Mpack ELK 6.3.2 without sudo elasticsearch_mpack-3.0.0.0-0.tar.gzIn the Parent Article I introduce the process to take an existing HortonWorks ELK Mpack and take it through a series of versions (up to 2.6) which would allow me to install the ELK Version and Components I want (ElasticSearch, Logstash, Kibana, FileBeats, MetricBeats) in Ambari 2.6. In this article I am going to install the last articles ELK 2.6 version into my local machine using Vagrant. With this working Test Base I will version the Mpack up to 3.0 as I change the files to allow install into HDP & HDF 3.0. I am also going to change to the current version of ELK 6.6.1.Starting with the 2.6 Mpack and the Ambari Quick Start Guide I am able to get a cluster installed very easily in my local machine. In my Test Base install I chose a single node

Easily convert any XML document to JSON format using TransformXML processor. Save following stylesheet in a file. Use TransformXML processor and specify xslt stylesheet. It will convert any XML document to JSON format.<?xml version="1.0"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:output method="text"/> <xsl:template match="/">{ <xsl:apply-templates select="*"/>} </xsl:template> <!-- Object or Element Property--> <xsl:template match="*"> "<xsl:value-of select="name()"/>" : <xsl:call-template name="Properties"/> </xsl:template> <!-- Array Element --> <xsl:template match="*" mode="ArrayElement"> <xsl:call-template name="Properties"/> </xsl:template> <!-- Object Properties --> <xsl:template name="Properties"> <xsl:variable name="childName" select="name(*[1])"/> <xsl:choos

MiniFi For Image Capture and Ingestion from Raspberry PIOur Apache NiFi 1.2 instance is waiting for data to be pushed from MiniFi instance.We export this to an XML template to convert to YAML.minifi-toolkit-1.0.2.1.4.0-5/bin/config.sh transform MinifiImage.xml config.ymlThen I scp over the YAML file to my MiniFi RPI 3.A clean running MiniFi instance. Not there will be some gzipped logs showing up in there.Grab all the images that are available (make sure they are complete and at least 30 seconds old and bigger than 4096 bytes).Schedule when you want things to run. Our use case is one picture a minute.Make sure you map your remote process group to the correct Input Port! MiniFi SetupMake sure you have Oracle's JDK 8 installed and in the path.export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/bin/minifi.sh startCheck the Logs tail -f log/minifi-app.logCheck the Statusbin/minifi.sh flowStatus controllerservices:health,bulletinsTest your Cameraraspistill -o cam4.jpgPython Code#!

I have been playing quite a bit with CDSW lately. Here is a quick article on how to setup a CDSW project in scala connecting to an external RDBMSStep 1: Create a new CDSW ProjectUsing the CDSW UI, create a new Scala Project:Step 2: Reference the external Jar in your spark-defaults.confOpen your project, and edit your spark-defaults.conf to add an external jar:spark.jars=http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.6/mysql-connector-java-5.1.6.jarStep 3: Create a simple Scala file to connect to the DBCreate a new file and add this code in it:val sqlContext = new org.apache.spark.sql.SQLContext(sc)val df = sqlContext.read.format("jdbc").option("url", "jdbc:mysql://[YOUR_SERVER_IP]:3306/[YOUR_DB]").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "[YOUR_TABLE]").option("user", "[YOUR_USER]").option("password", "[YOUR_PWD]").load()df.show()Step 4: Run your applicationLaunch a session and run your code:

IntroductionGeoMesa is an Apache licensed open source suite of tools that enables large-scale geospatial analytics on cloud and distributed computing systems, letting you manage and analyze the huge spatio-temporal datasets that IoT, social media, tracking, and mobile phone applications seek to take advantage of today.GeoMesa does this by providing spatio-temporal data persistence on top of the Accumulo, HBase, and Cassandra distributed column-oriented databases for massive storage of point, line, and polygon data. It allows rapid access to this data via queries that take full advantage of geographical properties to specify distance and area. GeoMesa also provides support for near real time stream processing of spatio-temporal data by layering spatial semantics on top of the Apache Kafka messaging system.GeoMesa features include the ability to:Store gigabytes to petabytes of spatial data (tens of billions of points or more)Serve up tens of millions of points in secondsIngest data faste

IntroductionInstalling GeoMesa HBase and GeoServer on HDPPrerequisitesHDP 2..6.4 on CentOS7.2Java Development Kit 1.8,Apache Maven,a GitHub client,and JAVA_HOME is configured below commands can be usedyum install git -y# configure JAVA_HOMEsed -i '$ a export JAVA_HOME=/usr/jdk64/jdk1.8.0_112' /etc/bashrcsed -i '$ a export PATH=$PATH:$JAVA_HOME/bin' /etc/bashrcsource /etc/bashrcjava -version#install Maven:cd /tmpwget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gztar -zxvf apache-maven-3.5.3-bin.tar.gzsudo mv /tmp/apache-maven-3.5.3 /optsudo chown -R root:root /opt/apache-maven-3.5.3sudo ln -s /opt/apache-maven-3.5.3 /opt/apache-mavenecho 'export PATH=$PATH:/opt/apache-maven/bin' | sudo tee -a /etc/bashrcsource /etc/bashrcmvn --versionDownload source codecd /tmpgit clone https://github.com/locationtech/geomesa.gitcd geomesa# Build geomesa-hbase from source codemvn clean install -pl geomesa-hbase/geomesa-hbase-dist -amcd /tmp/geomesa/geo

Kafka Broker:Java VersionWe recommend latest java 1.8 with G1 collector ( which is default in new version). If you are using Java 1.7 and G1 collector make sure you are on u51 or higher.A recommended setting for JVM looks like following-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80OS SettingsOnce the JVM size is determined leave rest of the RAM to OS for page caching. You need sufficient memory for page cache to buffer for the active writers and readers.In general disk throughput is a performance bottleneck and more disks are better. Depending on how one configures the flush behavior , a faster disk will be beneficial if they log.flush.interval.messages set to flush for every 100k messages or so.File Descriptors limits: Kafka needs open file descriptors for files and network connections . We recommend at least 128000 allowed for file descrip

Use CaseIoT Devices with Sensors, CamerasOverviewIn this, the third of the CDSW series, we build on using CDSW to classify images with a Python Apache MXNet model. In this use case we are receiving edge data from devices running MiniFi Agents that are collecting sensor data, images and also running edge analytics with TensorFlow. An Apache NiFi server collects this data with full data lineage using HTTP calls from the device(s). We then filter, transform, merge and route the sensor data, image data, deep learning analytics data and metadata to different data stores. As part of the flow we upload our images to a cloud hosted FTP server (could be S3 or any media store anywhere) and call a CDSW Model from Apache NiFi via REST and get the model results back as JSON. We are also storing our sensor data in Parquet files in HDFS. We then trigger a PySpark job from CDSW via API from Apache NiFi and check the status of that. We store the status result data in Parquet as well for PySp

This is an update to a previous article on accessing Philadelphia Open Crime Data and storing it in Apache Phoenix on HBase.Part 1: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.htmlFor nosql summithttps://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.htmlUpdate thishttp://princeton0.field.hortonworks.com:16010/master-statushttps://community.hortonworks.com/content/kbentry/54947/reading-opendata-json-and-storing-into-phoenix-tab.htmlPhilly Crime DataCity of PhillyApp Token76MVJDcTksxeS1uYPf8D0XdUFSecret TokenWZnlB_YJ5r9rjj_alWVdc_yqnxRpnIk5BHgbcrimehttps://data.phila.gov/resource/sspu-uyfa.json?$$app_token=76MVJDcTksxeS1uYPf8D0XdUF&$limit=5000https://www.opendataphilly.org/dataset/crime-incidents/resource/f6e5998d-9d33-4a45-8397-3a6bb8607d10https://www.opendataphilly.org/dataset/crime-incidentshttps://data.phila.gov/resource/sspu-uyfa.jsonhttps://data.phila.gov/resource/sspu-uyfa.jso

Tracking Air Quality with HDP and HDF: Part 2 - Indoor Air QualityUsing a few sensors on a MiniFi node we are able to generate some air quality sensor readings.Source: https://github.com/tspannhw/minifi-breakoutgarden/blob/master/aqminifi.pyData: row['bme680_tempc'] = '{0:.2f}'.format(sensor.data.temperature) row['bme680_tempf'] = '{0:.2f}'.format((sensor.data.temperature * 1.8) + 32) row['bme680_pressure'] = '{0:.2f}'.format(sensor.data.pressure) row['bme680_gas'] = '{0:.2f}'.format(gas) row['bme680_humidity'] = '{0:.2f}'.format(hum) row['bme680_air_quality_score'] = '{0:.2f}'.format(air_quality_score) row['bme680_gas_baseline'] = '{0:.2f}'.format(gas_baseline) row['bme680_hum_baseline'] = '{0:.2f}'.format(hum_baseline)See Part 1:https://community.hortonworks.com/articles/189630/tracking-air-quality-with-hdp-and-hdfi-part-1-apac.htmlNewark / NYC Hazecamhttps://hazecam.net/images/main/newark.jpgExample{"bme

Use Case: We have data stored in a MongoDB from a third party application in Amazon. Export from MongoDB to Parquet. Moving data from a single purpose data silo to your Enterprise Data Lake is a common use case. Using Apache NiFi we can easily save your data from this remote silo and bring it streaming into your analytics store for machine learning and deep analytics with Impala, Hive and Spark. It doesn't matter which cloud which are coming from or going to or from cloud to on-premise or various Hybrid situations. Apache NiFi will work in all of these situations which full data lineage and provenance on what it did when. I have created a mock dataset with Mockaroo. It's all about yummy South Jersey sandwiches. Our Easy MongoDB Flows to Ingest Mongo data to our Date Lake and another flow to load MongoDB. In our test, we loaded all the data from our Mock REST API into a MongoDB in the cloud. In the real world an application populated that dataset and now we need to bring it into our c

Series: Integration of Apache NiFi and Cloudera Data Science Workbench Part 2: Using Cloudera Data Science Workbench with Apache NiFi and Apache MXNet for GluonCV YOLO WorkloadsSummaryNow that we have shown it's easy to do standard NLP, next up is Deep Learning. As you can see NLP, Machine Learning, Deep Learning and more are all in our reach for building your own AI as a Service using tools from Cloudera. These can run in public or private clouds as scale. Now you can run and integrate machine learning services, computer vision APIs and anything you have created in house with your own Data Scientists or with the help of Cloudera Fast Forward Labs (https://www.cloudera.com/products/fast-forward-labs-research.html).The YOLO pretrained model will download the image to /tmp from the URL to process it. The Python 3 script will also download the GLUONCV model for YOLO3 as well.Using Pre-trained Model:yolo3_darknet53_voc Image Sourceshttps://github.com/tspannhw/images and/or https://p

The below steps describe how to change the Namenode log level while logged on as hdfs with the below steps, without the need to restart the namenodeGet the current log level$ hadoop daemonlog -getlevel {namenode_host}:50070BlockStateChangeDesired OutputConnecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange SubmittedLogName:BlockStateChangeLogClass: org.apache.commons.logging.impl.Log4JLoggerEffectiveLevel: INFOChange to DEBUG$ hadoop daemonlog -setlevel {namenode_host}:50070BlockStateChange DEBUG Desired OutputConnecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange&level=DEBUG SubmittedLogName:BlockStateChangeLogClass: org.apache.commons.logging.impl.Log4JLoggerSubmittedLevel: DEBUG SettingLevel to DEBUG ...EffectiveLevel: DEBUGValidate DEBUG mode$ hadoop daemonlog -getlevel {namenode_host}:50070BlockStateChangeDesired OutputConnecting to http://{namenode_host}:50070/logLevel?log=BlockStateChange SubmittedLogName:BlockStateChangeLogClass: org.ap

From Ambari 2.7 (after an ambari-server upgrade) it is required to backup / migrate and restore Infra Solr collection data.That is required because from Ambari 2.7 Infra Solr uses Solr 7 instead of Solr 5 and optimize command won't work on a Lucene 5 index, so it is required to do this offline.In order to not get stuck on any steps it is useful if we understands correctly what happens in the background as the documentation covers the happy path with the usage of /usr/lib/ambari-infra-solr-client/ambariSolrMigration.sh script. (https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-upgrade/content/backup_and_upgrade_ambari_infra_data.html). But fortunately every step can be done manually as well.Backup/Migration/Restore is only needed for Atlas and Ranger. Log Search will have a totally new schema, so there the Prerequisites: 1. First of all, upgrading the Ambari Infra Solr service is the next step right after Ambari server upgrade. (backup old Solr data then upgrade Infra S

In this tutorial, I create a NiFi cluster from the default blueprint provided with Cloudbreak 2.5.0 TP, but this post has been updated to reflect what is available in Cloudbreak 2.9.0.Update for Cloudbreak 2.9.0This post was originally written for Cloudbreak 2.5.0 TP, which introduced support for creating HDF Flow Management (NiFi) clusters. It has been updated to reflect the latest features available in Cloudbreak 2.9.0 general availability release.Cloudbreak 2.5.0 TP introduced support for creating HDF Flow Management (NiFi) clusters.The subsequent release, Cloudbreak 2.6.0 TP, introduced support for creating HDF Messaging Management (Kafka) clusters. Cloudbreak 2.7.0 GA introduced these as general availability features. In Cloudbreak 2.9.0 GA, two HDF 3.3 blueprints are included by default, one for Flow Management (NiFi) and one for Messaging Management (Kafka).What is Cloudbreak?Cloudbreak simplifies the provisioning, management, and monitoring of on-demand HDP and HDF clusters in

In this tutorial, I go through the steps of creating a data lake instance and creating a cluster attached to the data lake. I cover you how to use Cloudbreak web UI and CLI to perform these steps.Assumptions:These steps assume that you already have a running instance of Cloudbreak and that you have created a Cloudbreak credential that allows you to connect to your cloud provider.All the resources that I am creating in the tutorial are running on AWS. Whatever cloud provider you are using, I assume that you have an SSH key set up (if needed) and some network and subnet that you would like to use. Note that the data lake functionality cannot be used with OpenStack.The steps require that you create some resources (databases, LDAP, cloud storage) outside of Cloudbreak. See the "Meet the Prerequisites" section.This functionality is technical preview in Cloudbreak 2.9.x. It is not suitable for production.Update for Cloudbreak 2.9.0This post has been updated to reflect the latest functionalit

Cloudbreak 2.9.0 is now available! It is a general availability (GA) release, so - with an exception of some features that are marked as TP - it is suitable for production. Try it nowUpgrade to 2.9.0Quickly deploy by using quickstart on AWS, Azure, or Google CloudInstall manually on AWS, Azure, Google Cloud, or OpenStackNew featuresCloudbreak 2.9.0 introduces the following new features. While some of these features were introduced in Cloudbreak 2.8.0 TP, others are brand new:FeatureDescriptionDocumentationSpecifying resource group name on AzureWhen creating a cluster on Azure, you can specify the name for the new resource group where the cluster will be deployed. Resource group nameMultiple existing security groups on AWSWhen creating a cluster on AWS, you can select multiple existing security groups. This option is available only when an existing VPC is selected. Create a cluster on AWSEBS volume encryption on AWSYou can optionally configure encryption for EBS volumes attached to clus

How Does Tez determine the number of reducers? How can I control this for performance?In this article, I will attempt to answer this while executing and tuning an actual query to illustrate the concepts. Then I will provide a summary with a full explanation. if you wish, you can advance ahead to the summary.-------------0. Prep Work and Checklist We followed the Tez Memory Tuning steps as outlined in https://community.hortonworks.com/content/kbentry/14309/demystify-tez-tuning-step-by-step.htmlWe setup our environment, turning CBO and Vectorization On. set hive.support.sql11.reserved.keywords=false;set hive.execution.engine=tez;set hive.cbo.enable=true;set hive.compute.query.using.stats=true;set hive.stats.fetch.column.stats=true;set hive.stats.fetch.partition.stats=true;set hive.vectorized.execution.enabled=true;set hive.vectorized.execution.reduce.enabled = true;set hive.vectorized.execution.reduce.groupby.enabled = true;set hive.exec.parallel=true;set hive.exec.parallel.thread.num

Recently for a client I explored integrating nifi with Graylog. There is an AWS EC2 image for a full graylog install with the documentation for this to be found here http://docs.graylog.org/en/1.3/pages/installation/aws.html Configure a GELF HTTP connector to these settings http://docs.graylog.org/en/1.3/pages/sending_data.htmlMake a Python Script generating Dummy JSONS '{"short_message":"CHRIS HERE", "host":"example.org", "facility":"test", "_foo":"bar"}'Have a nifi HTTP post processor pointed at the Graylog inputConfirm messages have been received by graylog

Using Deployed Models as a Function as a Service Using Cloudera Data Science Workbench with Apache NiFi we can easily call functions within our deployed models from Apache NiFi as part of flows. I am working against CDSW on HDP (https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_hdp.html), but it will work for all CDSW regardless of install type.In my simple example, I built a Python model that uses TextBlob to run sentiment against a passed in sentence. It returns Sentiment Polarity and Subjectivity which we can immediately act upon in our flow.CDSW is extremely easy to work with and I was up and running in a few minutes. For my model, I created a python 3 script and a shell script for install details. Both of these artifacts are available here: https://github.com/tspannhw/nifi-cdswMy Apache NiFi 1.8 flow is here (I use no custom processors): cdsw-twitter-sentiment.xmlDeploying a Machine Learning Model as a REST ServiceOnce you login to CDSW

1. Now let's configure Knox to use our AD for authentication. Replace below content in Ambari > Knox > Config > Advanced topology<topology> <gateway> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>main.ldapRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> </param> <!-- changes for AD/user sync --> <param> <name>main.ldapContextFactory</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value&g

Introduction into Process Group Variables The ability configure variables at runtime in the UI was introduced in Apache NiFi 1.4 and will subsequently land in HDF 3.1. Variables can be referenced in property values of Processors and Controller Services that support expression language. This provides a centralized place to configure common values that may be referenced by multiple components and values that may be environmentally dependent. This guide provides a brief introduction to their usage. Configuration Variables are configured in a dialog that is accessed through the context menu. The menu item is available when the selection is a single Process Group. Below is an example of what the variable dialog would look like. The table will list all variables that are available in the scope of the current Process Group, the variable name, and variable value. Only variables defined in the current Process Group are editable. The table also contains a navigation link to go to the Process Gro

STOP Command::curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT http://172.26.78.29:8080/api/v1/clusters/Mycluster/hosts/prakash-ambariagent-node3/host_components/FLUME_HANDLER -d '{"RequestInfo":{"context":"Stop Flume","operation_level":{"level":"HOST_COMPONENT","cluster_name":"Mycluster","host_name":"prakash-ambariagent-node3","service_name":"FLUME"}},"Body":{"HostRoles":{"state":"INSTALLED"}}}'START Command:: curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT http://172.26.78.29:8080/api/v1/clusters/Mycluster/hosts/prakash-ambariagent-node3/host_components/FLUME_HANDLER -d '{"RequestInfo":{"context":"Start Flume","operation_level":{"level":"HOST_COMPONENT","cluster_name":"Mycluster","host_name":"prakash-ambariagent-node3","service_name":"FLUME"}},"Body":{"HostRoles":{"state":"STARTED"}}}'RESTART Command::curl --insecure -v -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://prakash-ambariserver-node1:8080/api/v1/clusters/Mycluster/requests -d '{"RequestInfo":{"

Log Log LogSudo logs have a lot of useful information on hosts, users and auditable actions that may be useful for cybersecurity, capacity planning, user tracking, data lake population, user management and general security.Symbol Model 1Step 1 - Get a FileStep 2 - Split Into LinesStep 3 - Set the Mime Type to Plain TextStep 4 - Extract Grok Step 5 - Action! Checking for More Options (All Named elements in the GrokPatterns)Example Sudo Log (could be /auth.log, /var/log/sudo.log, secure, ...)Jan 31 19:17:20 princeton0 su: pam_unix(su-l:session): session opened for user ambari-qa by (uid=0)Jan 31 19:17:20 princeton0 su: pam_unix(su-l:session): session closed for user ambari-qaJan 31 19:18:19 princeton0 su: pam_unix(su-l:session): session opened for user zeppelin by (uid=0)Jan 31 19:18:19 princeton0 su: pam_unix(su-l:session): session closed for user zeppelinJan 31 19:18:20 princeton0 su: pam_unix(su-l:session): session opened for user ambari-qa by (uid=0)Jan 31 19:18:20 princeton0 su: pam

Hive Pre Upgrade tool command fails after not able to access HDFS The command is not able to find the right configuration of hadoop core components, primarily due to unavailability of hadoop core component configuration files in the command. Resolution steps There are different ways to export the hadoop core configuration, one of them is below. 1. Follow the below HWX document to make sure you have properly followed the steps. (Check appropriate link for your version) https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/prepare_hive_for_upgrade.html 2. Follow till "Procedure for compacting Hive tables (no Kerberos)" step 3 and before running the pre-upgrade tool command, export hadoop and hive env filessource /etc/hadoop/conf/hadoop-env.sh source /etc/hive/conf/hive-env.sh 3. Now while running the pre-upgrade tool command, include two more paths in the command (/etc/hadoop/conf:/etc/hive/conf). So the command below similar to below. java