Resources

NiFi has previously supported the ability to refer to flow file attributes, system properties and environment properties within expression language (EL); however the community requested an enhancement to also support custom properties. This would give users even more flexibility either in processing, handling flow content, or even in flow configuration (e.g. referring to a custom property in EL for connection, server or service properties). In NiFi versions 0.7 & 1.0.0 an enhancement was added to allow administrators to define custom property files on nodes within their cluster and configure NiFi with their location so those properties could be loaded and available within EL. A new field in the nifi.properties file ( nifi.variable.registry.properties) is available for an administrator to set the paths of one or more custom properties files for use by NiFi. Figure 1 - Custom Properties reference in nifi.properties Once the nifi.properties file is updated custom attribute

IMPORTANT:DO NOT FORGET to backup your Ambari database before executing DELETE API calls. This is tested on Ambari version 2.6.x & 2.7.1. Kindly use these steps on your Test cluster first OR Contact Cloudera Support for more details.Step 1: Take a backup of your Ambari database# mkdir /var/tmp/postgres_backup # pg_dump -U ambari ambari > /var/tmp/postgres_backup/$(date +"%Y%m%d%H%M%S")_ambari-bkp.sqlStep 2: Install jq package (If not available)# rpm -qa | grep jq# yum whatprovides jq# yum install jq -yStep 3: Get All (AD/LDAP) Users & Groups# curl --insecure -u admin:admin -H 'X-Requested-By: ambari' -X GET http://$(hostname -f):8080/api/v1/users?Users/ldap_user=true | jq -r '.items[].Users.user_name' > ambari-ldap-users.txt# curl --insecure -u admin:admin -H 'X-Requested-By: ambari' -X GET http://$(hostname -f):8080/api/v1/groups?Groups/ldap_group=true | jq -r '.items[].Groups.group_name' > ambari-ldap-groups.txtStep 4: Verify Users & Groups that needs to

Pre-requisites:1. Install Docker in all nodemanager hosts and configure:It is recommended to install the version of Docker that is provided by your Operating System vendor. The Docker package has been known by several names; docker-engine, docker, and docker-ce.yum install dockerIf having issues installing docker, following document can be followed:https://docs.docker.com/install/Edit ‘/etc/docker/daemon.json’ and add the following options:{        "live-restore" : true,        "debug" : true,        "dns": ["<YARN registry dns ip addr>"] }If not using HTTPS, Configure each of the cluster hosts to skip HTTPS checks by adding following line in ‘/etc/docker/daemon.json’"insecure-registries": ["<docker registry server>:5000"]2. Create a private local docker registry(Optional)2.1- Designate a server in the cluster for use by the Docker registry. Minimal resources are required, but sufficient disk space i

Pre-requisite:1. Setup Ambari with LDAP and Sync. 2. Setup Knox and point to same LDAP as Ambari server.Enable SSO for Ambari:1. Get the Knox public cert by running below.openssl s_client -connect KNOXHOST:8443 <<<'' | openssl x509 -out /tmp/knox.crt2. Run "ambari-server setup-sso"3. "provider URL": Enter https://<hostname>:8443/gateway/knoxsso/api/v1/websso4. "Public Certificate pem" : Provide step1 cert file content without BEGIN/END blocks. -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- Note: Make sure your /etc/ambari-server/conf/jwt-cert.pem file should have only one BEGIN/END5. You can select default for rest of the configs.6. Re-start Ambari server:Knox Configurations1. If Ambari and Knox is in different host then Whitelist Ambari URL. In Advanced knoxsso-topology modify below config for whitelisting all (or you can write regex for specific) <param> <name>knoxsso.redirect.whitelist.regex</name> <value

Using Raspberry Pi 3B+ with Apache NiFi MiNiFi and Google Coral Accelerator and Pimoroni Inky PhatFirst we need to unbox our new goodies. The Inky Phat is an awesome E-Ink display with low power usage that stays displayed after shutdown! Next I added a new Google Coral Edge TPU ML Accelerator USB Coprocessor to a new Raspberry Pi 3B+. This was so easy to integrate and get up and running.Let's unbox this beautiful device (but be careful when it runs it can get really hot and there is a warning in the instructions). So I run this on top of an aluminum case and with a big fan on it. Pimoroni Inky PhatIt is pretty easy to set this up and it provides a robust Python library to write to our E-Ink display. You can see an example screen here.https://github.com/pimoroni/inkyPimoroni Inky pHAT ePaper eInk Display in RedPimoroni Inky Phat (Red)https://shop.pimoroni.com/products/inky-phathttps://github.com/pimoroni/inkyhttps://pillow.readthedocs.io/en/stable/reference/ImageDraw.htmlh

Hi All, Hope you are all doing good!!! I have 8+ years experiance in IT infrastraucture support. Noe I planned to switch my carrer to Hadoop Administration. So, I started learning things around Big Data and found Cloudera would be the best choice. I would like to whuch topic have to follow to get better knowledge in Hadoop Administraton. Adhav 

Hi,  How do I delete my support account?  I don't see any options to delete my account.   

Error:Error: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 configs from ZooKeeper (state=08S01,code=0)Issue: Beeline connection failing using Kerberos principal and ServiceDiscoveryMode beeline -u "jdbc:hive2://zk1.com:2181,zk2.com:2181,zk3.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;auth=KERBEROS;principal=myuser/hiverserver2@EXAMPLE.COM;httpPath=cliservice;saslQop=auth-conf" Resolution: We do not need to pass the kerberos principal within the beeline command because we are using zookeeper. You must already have valid Kerberos ticket.Example:kinit myuserbeeline -u "jdbc:hive2://zk1.com:2181,zk2.com:2181,zk3.com:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=binary;httpPath=cliservice;saslQop=auth-conf" Note: You must have a valid ticket first. If no valid ticket is present you will see an error like the following:The error in the log "echanism level: Fail

Ambari is the heart of any HDP cluster. It provides us the feature of provisioning, managing, monitoring and securing Hadoop / HDP clusters. It's is a Java program which interacts with Database to read the cluster details and runs on embedded jetty server. Many times we find issues with Ambari server performance. It's Ambari UI operations sometimes responds slowly or the startup might take longer time if it is not properly tuned. So in order to troubleshoot the ambari server performance related issues we should look at some of the data/stats and tuning parameters to make the ambari server perform better. In this article we will talk about some very basic tuning parameters and performance related troubleshooting. What all information's needed? When we notice that the ambari server is responding slow then we should look first the following details first: 1). The number of hosts added to the ambari cluster. So that accordingly we can tune the ambari agent thread pools. 2). The number of c

Hi,If you don't want SMARTSENSE in your cluster but still it comes as default selected component during install wizard then go through below steps to save yourself some trouble.Tried on HDP Version : 3.0 and 3.1Goto below path on the ambari-server node:/var/lib/ambari-server/resources/stacks/HDP/3.0/services/SMARTSENSE/metainfo.xmlOpen the above file in editor mode (e.g. vi)Uncomment or delete the below line [Line 23 may vary in different release]<selection>MANDATORY</selection>After making the above change restart ambari-server and proceed with cluster install wizard.Now SMARTSENSE won't be a mandate component.Thanks for reading.

https://youtu.be/-HMyEpDJeGgConfiguring Ambari 2.4.2 and HDP 2.5 for Kerberos using AD as the KDC Add bonus coverage of adding a new datanode to a HDP cluster that is secured. There are empty OUs created in AD to store hadoop principals/hadoop nodes (HadoopServices)Hadoopadmin user has administrative credentials with delegated control of "Create, delete, and manage user accounts" on above OUDelegate OU permissions to hadoopadmin for OU=HadoopServices. In 'Active Directory Users and Computers' app:right click HadoopServicesDelegate ControlNextAddhadoopadminchecknamesOKSelect "Create, delete, and manage user accounts"OKKDC:KDC host: ad01.prod.hortonworks.netRealm name: PROD.HORTONWORKS.NETLDAP url: ldaps://ad01.prod.hortonworks.netContainer DN: OU=HadoopServices,DC=prod,DC=hortonworks,DC=netDomains: prod.hortonworks.netKadmin:Kadmin host: ad01.prod.hortonworks.netAdmin principal: hadoopadmin@PROD.HORTONWORKS.NETAdmin password: xxxxxx

I was recently tinkering with the walmart rest-api. This is publicly available interface and can be used for a quick price look up for products. The overall goal of the project is to keep track of the cost of specific shopping carts day to day but this intermediate step provides an interesting example-case.The requirements of this stage:Use UPC codes to provide a lookupAvoid having the pass the walmart API key to the internal client making the callExtract features such as price in preparation for a JDBC database entry.Core Concepts in NifiNifi has the ability to serve as a custom restful api which is managed with the "HandleHttpRequest" and "HandleHttpResponce" processors. This can be a Get/Post or any of the other common typesNifi can make calls to an external rest API via the "InvokeHTTP" ProcessorXML Data can be extracted with the "EvaluateXPath" ProcessorThe HandleHttpRequest ProcessorThis processor receives the incoming rest call and makes a flow file with information pertainin

OverviewIn this article I show how to very quickly build custom logging against source log files. Adding new custom logging to the flow takes only seconds: each new custom log is copy-pasted from an existing one and 3 configuration properties are changed. The result is a single flow with many diverse custom logs against your system.Use CasesUse cases include:Application development: permanent or throway logging focused on debugging software issues or NiFi flows.Production: logging for focused metrics gathering, audting, faster troubleshooting, alertingAll envts: prefiltering Splunk ingests to save on cost and filtering out unneeded log dataUse your imagingation … how can this flow pattern make your life better / more effective?Flow DesignSingle customized logA single customized log is achieved by tailing a log file, splitting the lines of text, extracting lines that match a regex, specifiying a custom log filename for the regex-extracted lines, and appending the lines to that custom l

This is a Work In Progress Article. Downloads:HDP/HDF 3 Ambari 2.7 Mpack ELK 6.3.2 with sudo (for non root user install) elasticsearch_mpack-3.0.0.0-1.tar.gzHDP/HDF 3 Ambari 2.7 Mpack ELK 6.3.2 without sudo elasticsearch_mpack-3.0.0.0-0.tar.gzIn the Parent Article I introduce the process to take an existing HortonWorks ELK Mpack and take it through a series of versions (up to 2.6) which would allow me to install the ELK Version and Components I want (ElasticSearch, Logstash, Kibana, FileBeats, MetricBeats) in Ambari 2.6. In this article I am going to install the last articles ELK 2.6 version into my local machine using Vagrant. With this working Test Base I will version the Mpack up to 3.0 as I change the files to allow install into HDP & HDF 3.0. I am also going to change to the current version of ELK 6.6.1.Starting with the 2.6 Mpack and the Ambari Quick Start Guide I am able to get a cluster installed very easily in my local machine. In my Test Base install I chose a single node

Easily convert any XML document to JSON format using TransformXML processor. Save following stylesheet in a file. Use TransformXML processor and specify xslt stylesheet. It will convert any XML document to JSON format.<?xml version="1.0"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:output method="text"/> <xsl:template match="/">{ <xsl:apply-templates select="*"/>} </xsl:template> <!-- Object or Element Property--> <xsl:template match="*"> "<xsl:value-of select="name()"/>" : <xsl:call-template name="Properties"/> </xsl:template> <!-- Array Element --> <xsl:template match="*" mode="ArrayElement"> <xsl:call-template name="Properties"/> </xsl:template> <!-- Object Properties --> <xsl:template name="Properties"> <xsl:variable name="childName" select="name(*[1])"/> <xsl:choos

MiniFi For Image Capture and Ingestion from Raspberry PIOur Apache NiFi 1.2 instance is waiting for data to be pushed from MiniFi instance.We export this to an XML template to convert to YAML.minifi-toolkit-1.0.2.1.4.0-5/bin/config.sh transform MinifiImage.xml config.ymlThen I scp over the YAML file to my MiniFi RPI 3.A clean running MiniFi instance. Not there will be some gzipped logs showing up in there.Grab all the images that are available (make sure they are complete and at least 30 seconds old and bigger than 4096 bytes).Schedule when you want things to run. Our use case is one picture a minute.Make sure you map your remote process group to the correct Input Port! MiniFi SetupMake sure you have Oracle's JDK 8 installed and in the path.export JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/bin/minifi.sh startCheck the Logs tail -f log/minifi-app.logCheck the Statusbin/minifi.sh flowStatus controllerservices:health,bulletinsTest your Cameraraspistill -o cam4.jpgPython Code#!

I have been playing quite a bit with CDSW lately. Here is a quick article on how to setup a CDSW project in scala connecting to an external RDBMSStep 1: Create a new CDSW ProjectUsing the CDSW UI, create a new Scala Project:Step 2: Reference the external Jar in your spark-defaults.confOpen your project, and edit your spark-defaults.conf to add an external jar:spark.jars=http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.6/mysql-connector-java-5.1.6.jarStep 3: Create a simple Scala file to connect to the DBCreate a new file and add this code in it:val sqlContext = new org.apache.spark.sql.SQLContext(sc)val df = sqlContext.read.format("jdbc").option("url", "jdbc:mysql://[YOUR_SERVER_IP]:3306/[YOUR_DB]").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "[YOUR_TABLE]").option("user", "[YOUR_USER]").option("password", "[YOUR_PWD]").load()df.show()Step 4: Run your applicationLaunch a session and run your code:

IntroductionGeoMesa is an Apache licensed open source suite of tools that enables large-scale geospatial analytics on cloud and distributed computing systems, letting you manage and analyze the huge spatio-temporal datasets that IoT, social media, tracking, and mobile phone applications seek to take advantage of today.GeoMesa does this by providing spatio-temporal data persistence on top of the Accumulo, HBase, and Cassandra distributed column-oriented databases for massive storage of point, line, and polygon data. It allows rapid access to this data via queries that take full advantage of geographical properties to specify distance and area. GeoMesa also provides support for near real time stream processing of spatio-temporal data by layering spatial semantics on top of the Apache Kafka messaging system.GeoMesa features include the ability to:Store gigabytes to petabytes of spatial data (tens of billions of points or more)Serve up tens of millions of points in secondsIngest data faste

IntroductionInstalling GeoMesa HBase and GeoServer on HDPPrerequisitesHDP 2..6.4 on CentOS7.2Java Development Kit 1.8,Apache Maven,a GitHub client,and JAVA_HOME is configured below commands can be usedyum install git -y# configure JAVA_HOMEsed -i '$ a export JAVA_HOME=/usr/jdk64/jdk1.8.0_112' /etc/bashrcsed -i '$ a export PATH=$PATH:$JAVA_HOME/bin' /etc/bashrcsource /etc/bashrcjava -version#install Maven:cd /tmpwget http://mirror.jax.hugeserver.com/apache/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gztar -zxvf apache-maven-3.5.3-bin.tar.gzsudo mv /tmp/apache-maven-3.5.3 /optsudo chown -R root:root /opt/apache-maven-3.5.3sudo ln -s /opt/apache-maven-3.5.3 /opt/apache-mavenecho 'export PATH=$PATH:/opt/apache-maven/bin' | sudo tee -a /etc/bashrcsource /etc/bashrcmvn --versionDownload source codecd /tmpgit clone https://github.com/locationtech/geomesa.gitcd geomesa# Build geomesa-hbase from source codemvn clean install -pl geomesa-hbase/geomesa-hbase-dist -amcd /tmp/geomesa/geo

Kafka Broker:Java VersionWe recommend latest java 1.8 with G1 collector ( which is default in new version). If you are using Java 1.7 and G1 collector make sure you are on u51 or higher.A recommended setting for JVM looks like following-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80OS SettingsOnce the JVM size is determined leave rest of the RAM to OS for page caching. You need sufficient memory for page cache to buffer for the active writers and readers.In general disk throughput is a performance bottleneck and more disks are better. Depending on how one configures the flush behavior , a faster disk will be beneficial if they log.flush.interval.messages set to flush for every 100k messages or so.File Descriptors limits: Kafka needs open file descriptors for files and network connections . We recommend at least 128000 allowed for file descrip

Use CaseIoT Devices with Sensors, CamerasOverviewIn this, the third of the CDSW series, we build on using CDSW to classify images with a Python Apache MXNet model. In this use case we are receiving edge data from devices running MiniFi Agents that are collecting sensor data, images and also running edge analytics with TensorFlow. An Apache NiFi server collects this data with full data lineage using HTTP calls from the device(s). We then filter, transform, merge and route the sensor data, image data, deep learning analytics data and metadata to different data stores. As part of the flow we upload our images to a cloud hosted FTP server (could be S3 or any media store anywhere) and call a CDSW Model from Apache NiFi via REST and get the model results back as JSON. We are also storing our sensor data in Parquet files in HDFS. We then trigger a PySpark job from CDSW via API from Apache NiFi and check the status of that. We store the status result data in Parquet as well for PySp

This is an update to a previous article on accessing Philadelphia Open Crime Data and storing it in Apache Phoenix on HBase.Part 1: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.htmlFor nosql summithttps://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.htmlUpdate thishttp://princeton0.field.hortonworks.com:16010/master-statushttps://community.hortonworks.com/content/kbentry/54947/reading-opendata-json-and-storing-into-phoenix-tab.htmlPhilly Crime DataCity of PhillyApp Token76MVJDcTksxeS1uYPf8D0XdUFSecret TokenWZnlB_YJ5r9rjj_alWVdc_yqnxRpnIk5BHgbcrimehttps://data.phila.gov/resource/sspu-uyfa.json?$$app_token=76MVJDcTksxeS1uYPf8D0XdUF&$limit=5000https://www.opendataphilly.org/dataset/crime-incidents/resource/f6e5998d-9d33-4a45-8397-3a6bb8607d10https://www.opendataphilly.org/dataset/crime-incidentshttps://data.phila.gov/resource/sspu-uyfa.jsonhttps://data.phila.gov/resource/sspu-uyfa.jso

Tracking Air Quality with HDP and HDF: Part 2 - Indoor Air QualityUsing a few sensors on a MiniFi node we are able to generate some air quality sensor readings.Source: https://github.com/tspannhw/minifi-breakoutgarden/blob/master/aqminifi.pyData: row['bme680_tempc'] = '{0:.2f}'.format(sensor.data.temperature) row['bme680_tempf'] = '{0:.2f}'.format((sensor.data.temperature * 1.8) + 32) row['bme680_pressure'] = '{0:.2f}'.format(sensor.data.pressure) row['bme680_gas'] = '{0:.2f}'.format(gas) row['bme680_humidity'] = '{0:.2f}'.format(hum) row['bme680_air_quality_score'] = '{0:.2f}'.format(air_quality_score) row['bme680_gas_baseline'] = '{0:.2f}'.format(gas_baseline) row['bme680_hum_baseline'] = '{0:.2f}'.format(hum_baseline)See Part 1:https://community.hortonworks.com/articles/189630/tracking-air-quality-with-hdp-and-hdfi-part-1-apac.htmlNewark / NYC Hazecamhttps://hazecam.net/images/main/newark.jpgExample{"bme

Use Case: We have data stored in a MongoDB from a third party application in Amazon. Export from MongoDB to Parquet. Moving data from a single purpose data silo to your Enterprise Data Lake is a common use case. Using Apache NiFi we can easily save your data from this remote silo and bring it streaming into your analytics store for machine learning and deep analytics with Impala, Hive and Spark. It doesn't matter which cloud which are coming from or going to or from cloud to on-premise or various Hybrid situations. Apache NiFi will work in all of these situations which full data lineage and provenance on what it did when. I have created a mock dataset with Mockaroo. It's all about yummy South Jersey sandwiches. Our Easy MongoDB Flows to Ingest Mongo data to our Date Lake and another flow to load MongoDB. In our test, we loaded all the data from our Mock REST API into a MongoDB in the cloud. In the real world an application populated that dataset and now we need to bring it into our c

Series: Integration of Apache NiFi and Cloudera Data Science Workbench Part 2: Using Cloudera Data Science Workbench with Apache NiFi and Apache MXNet for GluonCV YOLO WorkloadsSummaryNow that we have shown it's easy to do standard NLP, next up is Deep Learning. As you can see NLP, Machine Learning, Deep Learning and more are all in our reach for building your own AI as a Service using tools from Cloudera. These can run in public or private clouds as scale. Now you can run and integrate machine learning services, computer vision APIs and anything you have created in house with your own Data Scientists or with the help of Cloudera Fast Forward Labs (https://www.cloudera.com/products/fast-forward-labs-research.html).The YOLO pretrained model will download the image to /tmp from the URL to process it. The Python 3 script will also download the GLUONCV model for YOLO3 as well.Using Pre-trained Model:yolo3_darknet53_voc Image Sourceshttps://github.com/tspannhw/images and/or https://p