Community Articles

Goal: Ingest Geospatial data (ais data in CSV format) by using GeoMesa HBase after GeoMesa HBase is installed and configured. Follow install-and-configure-geomesa-hbase-and-geoserver if GeoMesa HBase is not installed151001000015660,1001001,-1,55.476868,-25.874775,0.4,50,-999.9,511,2015-10-01 06:14:33,1151001000015661,1001001,-1,55.476868,-25.874775,0.4,50,-999.9,511,2015-10-01 06:14:34,1151002008706375,1001001,-1,55.82175,-26.442037,0.7,40,-999.9,511,2015-10-02 05:12:46,1151002008706376,1001001,-1,55.82175,-26.442037,0.7,40,-999.9,511,2015-10-02 05:12:47,1Create GeoMesa SimpleFeatureType (sft) configuration file as below in ais_data_conf.sftgeomesa.sfts.ais-data = { attributes = [ { name = "arkposid", type = "Long", index = true } { name = "mmsi", type = "Integer", index = false } { name = "navigationalstatus", type = "Integer", index = false } { name = "coords", type = "Point", index = true, srid = 4326, default = true

Vision Thing: Part 2: Processing, Capturing and Displaying Live Image Feeds with Apache NiFi, MiniFi, OpenCV, Python, MXNetAs part of processing live webcam images from devices I want to display the last one ingested to see what's going on. Those same images run through Apache MXNet, NVidia TensorRT and OpenCV for analysis.I wanted to list all the images that I have stored on my NiFi server sent by Jetson TX1 device. So I went old school and have Apache NiFi server up some simple CGI script. It lists images and wraps them in HTML.index.sh ls /opt/demo/images2/ | /opt/demo/buildpage.shbuildpage.sh#!/bin/shecho '<html><head><meta http-equiv="refresh" content="60"> <title>NiFi List Images</title> </head> <body> <p> <br> <b>List Images</b> <br><br>'sed 's/^.*/<a target="_new" href="http:\/\/princeton1\.field\.hortonworks\.com\:9099\?img_name\=&">&<\/a><br\/>/'echo '</body>&

NOTE: This was recorded before everything kicked off with Facebook and Cambridge Analytica. Interesting timing.https://roaringelephant.org/2018/03/27/episode-80-big-data-tracking/ Last June, Wolfie Christl published a 93 page report Corporate Surveillance in Everyday Life using big data tracking. Apart from the massive pdf that can be downloaded on the net, an extensive summary can be found on the Cracked Labs website.In this episode we go over the content and give our views on the subject.Podcast: Play in new window | Download (Duration: 51:25 — 35.6MB)If you want to follow along with us while we are discussing the different point in the onlin earticle, here is the link: http://crackedlabs.org/en/corporate-surveillancePlease use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

Open Source Computer Vision with TensorFlow, Apache MiniFi, Apache NiFi, OpenCV, Apache Tika and PythonIn preparation for this talk, I am releasing some articles detailing how to work with images. In this one I use my custom Web Camera processor to ingest web camera images via Apache NiFi on an OSX laptop equipped with two webcameras.This will let you ingest as many images as you need for security, motion detection or fun.Custom Processor:https://github.com/tspannhw/GetWebCameraMy processor is a thin wrapper on an awesome Java library: http://webcam-capture.sarxos.pl/See: https://github.com/sarxos/webcam-capture/blob/master/webcam-capture/src/example/java/TakePictureExample.javaTo use the processor, you can clone my github project or download the prebuilt NAR and install to your nifi/lib directory and restart your NiFi server.It is easy to use. Add the processor to your workflow at the start to ingest images from your webcam.The processor takes two properties: imagefilename and

AD (After Druid)In my opinion, Druid creates a new analytic service that I term the real-time EDW. In traditional EDWs the process of getting from the EDW to an optimized OLAP could take hours depending on the size of the EDW. Having a cube take six or more hours was not unusual for a lot of companies. The process also tended to be brittle and needed to be closely monitored.In addition to real-time, Druid also facilitates long-term analytics which effectively provides a Lambda architecture for your data warehouse. Data streams into Druid and is held in-memory for a configurable amount of time. While in-memory the data can be queried and visualized. After a period of time the data is then passed to long-term (historical) storage as segments on HDFS. These segments can also be part of the same visualization as the real-time data. As mentioned previously, all data in Druid contains a timestamp. The other data elements consist of the same properties as traditional EDWs: dimensions and mea

Good design goes mostly unnoticed. If a user interface works well, you simply complete a task and move on to the next. That time when something doesn't work as expected, or you think why do I have to do it this way, is when the design becomes much more noticeable. The following is a look back over recent months at some notable UI/UX improvements that have recently made or will be soon making their way to Apache NiFi. They vary a lot in size and complexity, but overall they contribute equally to make NiFi better by design.More Control Over Restricted Components New access policies were introduced for more fine-grain control of user interactions with restricted components. For example, permissions can be granted around reading of or writing to a file system, executing code, and keytab file access. Good to know – all new policies introduced are subcategories of the one previously in place, so a user holding a restricted component policy will receive all new ones. Administrators wi

Druid is an OLAP solution for streaming event data as well as OLAP for long-term storage. All Druid data requires a timestamp. Druid’s storage architecture is based off the timestamp similar to how HBase stores by key. Following are some key benefits of Druid:Real-time EDW on event data (time-series)Long-term storage leveraging HDFSHigh availabilityExtremely performant querying over large data setsAggregation and IndexingHigh-level of data compressionHive integrationDruid provides a specific solution for specific problems that could not be handled by any other technology. With that being said, there are instances where Druid may not be a good fit:Data without a timestampNo need for real-time streamingNormalized (transactional) data (no joins in Druid)Small data setsNo need for aggregating measuresNon-BI queries like Spark or streaming lookupsWhy Druid?BD (Before Druid)In traditional EDWs data is broken into dimensional tables and fact tables. Dimensions describe an object. For example

Now that we have the baseline configuration in place for KnoxSSO, as documented in part I of the series, we are ready to configure single sign-on for Ranger. This is a nice convenience for administrators that live in both Ambari and Ranger as part of their daily platform activities. It is also a security improvement relative to local passwords.To configure Ranger for KnoxSSO, we'll need the public key for Knox. Recall that there are a few ways to obtain this, we'll use the following.openssl s_client -connect ${knoxserver}:8443 < /dev/null | openssl x509 -out /tmp/knox.crtYou'll want to copy the base64 data between BEGIN CERTIFICATE and END CERTIFICATE lines.We're now ready to configure Ranger:All we need to configure is the Knox SSO provider URL and the SSO public key. The SSO public key is the copied data we just discussed from the Knox certificate.The SSO provider URL is the URL we configured in part I that corresponds to the Knox SSO topology.Now let's try to log in to Ranger usi

In this article we will see how to produce messages using a simple python script and consume messages using ConsumeMQTT processor and put them in HDFS using PutHDFSNote: I'm using CentOS 7 and HDP 2.6.3 for this article.1) Install MQTT sudo yum -y install epel-releasesudo yum -y install mosquitto.2) Start MQTT sudo systemctl start mosquittosudo systemctl enable mosquitto.3) Install paho-mqtt python libraryyum install python-pippip install paho-mqtt.4) Configure MQTT password for the user.I have created a sample user 'aditya' and set the password to 'test'[root@test-instance-4 ~]# useradd aditya[root@test-instance-4 ~]# sudo mosquitto_passwd -c /etc/mosquitto/passwd adityaPassword:Reenter password:.5) Disable anonymous login to MQTT Open the file (/etc/mosquitto/mosquitto.conf ) and add the below entries and restart mosquittoallow_anonymous falsepassword_file /etc/mosquitto/passwd sudo systemctl restart mosquitto.6) Design the NiFi flow to consume messages and put into hdfsConfigure MQT

There are many ways to integrate Apache NiFi and Apache Spark. We can call Apache Spark Streaming via S2S (Apache NiFi's Site to Site) or Kafka. If you want to execute a regular Apache Spark job, you can do that via Apache Livy which is included in HDP 2.6+. This is how Apache Zeppelin integrates with Apache Spark, so it's secure and a reasonable approach.I use this approach when I want to use Spark to process part of my process in the middle of an Apache NiFi flow.Syntax for Calling a Job This job is stored in HDFS as /apps/logs*jar with the class name com.dataflowdeveloper.logs.Logs. Schema For Apache Livy Status Messages { "type": "record", "name": "livystatus", "fields": [ { "name": "id", "type": [ "null", "int" ] }, { "name": "state", "type": [ "null", "string" ] }, { "name": "appId", "type": [ "null", "string" ] }, { "name": "driverLogUrl", "type": [ "null", "string" ] }, { "name": "sparkUiUrl", "type": [ "null

Introduction:When I start thinking about a leading opensource cloud computing platform, Openstack comes into a picture. OpenStack controls a large pools of compute, storage, and networking resources throughout a datacenter. Openstack has gained a very high popularity in the technology market. It introduces a new services and features in every release cycle to address critical IT requirements. A very important requirement for any IT organization is to build a robust platform to perform data analytics with a large data sets. BigData is the latest buzzword in the IT Industry. This article primarily focusses on how Openstack plays a key role in addressing BigData usecase.Bigdata on Openstack!Now a days the data is generated everywhere and its volume is growing exponentially. Data is generated from web server, application server, database server in the form of user information, log files and system state information. Apart from this, a huge volume of data is generated from IoT devices like

What is Apache Ambari?Apache Ambari is an open-source software to install, manage and monitor Apache Hadoop family of components. It automates many of the basic actions performed and provides a simple and easy to use UI.How does Ambari work?Hadoop and its ecosystem of software are typically installed as a multi-node deployment. Ambari has a two level architecture of an Ambari Server and an Ambari agent. Ambari Server centrally manages all the agents and sends out operations to be performed on individual agents. Agents are installed by the server on each node (host) which in turn installs, configures and manages services in the agentWhat are Services?Services are the various components of the Hadoop ecosystem such as HDFS, YARN, Hive, HBase, Oozie, Druid, etc. One of the most popular open-source Hadoop distributions is the Hortonworks Data Platform (HDP)How is a stack like HDP installed by Ambari? Each version of HDP corresponds to a version of Ambari which supports the HDP version.The

Integration Apache OpenNLP 1.8.4 into Apache NiFi 1.5 For Real-Time Natural Language Processing of Live Data StreamsThis is an update to the existing processor. This one seems to work better and faster.VersionsApache OpenNLP 1.8.4 with Name, Location and Date Processing.I also improved the output format and added Date parsing.Example Outputnlp_location_1 Chinanlp_name_1 Andrew TurnerReleasehttps://github.com/tspannhw/nifi-nlp-processor/releases/tag/2.0InstallationDownload NAR here: https://github.com/tspannhw/nifi-nlp-processor/releases/tag/2.0Install nar file to /usr/hdf/current/nifi/lib/Create a model directory with permissions for nifi userDownload models (see below)Restart Apache NiFi via AmbariDownload Modelswget http://opennlp.sourceforge.net/models-1.5/en-ner-date.binwget http://opennlp.sourceforge.net/models-1.5/en-ner-location.binwget http://opennlp.sourceforge.net/models-1.5/en-ner-money.binwget http://opennlp.sourceforge.net/models-1.5/en-ner-organization.binwget http://op

Integrating TensorFlow 1.6 Image Labelling with HDF 3.1 and Apache NiFi 1.5 This is community Apache NiFi custom processor that I have written with help from Simon Ball. This is after his original change, I converted the results to a different style and incorporated changes to Google's TensorFlow Label_Image in Java example. Images Supported as Flow Files JPG, PNG, GIF Versions Updated TensorFlowProcessor to TF 1.6. Summary I added more tests, did more clode cleanup, change the top 5 returned with cleaner naming. This processor has been tested in a few environments, please give it a try and let me know your results. The performance and stability seem quite good, hopefully enough for your use cases. You can also use the MiniFi C++ Agent's built-in C++ TensorFlow support if you wish to have fast local TensorFlow at the edge. Video JUNit Test for TensorFlow Processor Here is another flow where I am send an image to

In this article, I'll show how to analyze a real-time data stream using Spark Structured Streaming. I wanted to provide a quick Structured Streaming example that shows an end-to-end flow from source (Twitter), through Kafka, and then data processing using Spark. To accomplish this, I used Apache NiFi (part of Hortonworks HDF) to capture the Twitter data and send it to Apache Kafka. From here, Spark was used to consume each Twitter payload (as JSON), parse, and analyze the data in real-time. Before I jump into the technical details, it's good to understand some of the business value of this process. There are many practical applications based on this technology stack as well as the code that I describe below. Here are a few of the business applications that can be implemented using this technology: Security: Real-time log analysis allows organizations to extract IP addresses, ports, services and events. This data can then be aggregated based on a time window, analyzed, and monitored for

PROBLEM:Changed Ambari from root to non-root user and hiveview doesn't work. Ambari hive view fails with Usernames not matched error. ERROR: Service 'userhome' check failed:java.io.IOException: Usernames not matched: name=ambari-server-<cluster_name> != expected=<custom_user> at sun.reflect.GeneratedConstructorAccessor243.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)ROOT CAUSE: Configuration issue on Ambari Hive view definition as username doesn't match.SOLUTION: Navigate to Hive and YARN Configs in Ambari UI and change as below and restart respective services. A) Custom webhcat-site webhcat.proxyuser.<AMBARI_SERVER_PRINCIPAL_USER>.groups=*webhcat.proxyuser.<AMBARI_SERVER_PRINCIPAL_USER>.hosts=* B) Custom yarn-sit

###Finding Stale Configs- We can use the Ambari APIs to find the stale configurations. (Notice: ?HostRoles/stale_configs=true)http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true- If there is no stale configuration present in the cluster then we might see the items as empty "[]". Output may be as following:{ "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true", "items" : [ ]}.- Now i intentionally made some changes to the "Advanced core-site.xml" from ambari and added the following properties to it, Just to see if it changes the stale configuration list.hadoop.proxyuser.joy.hosts=*hadoop.proxyuser.joy.groups=*.- As we made some changes and the affected components are not yet restarted hence we will see that there are few stale configuration present in the cluster then we might see the output something like following (Notice: "stale_configs" : true ){ "href" : "http://erie1

In Zeppelin 0.7.0 Document, it is said that library should use 0.13.Reference : How to execute HIVE LLAP queries from Zeppelin in HDP 2.6.1Hive Interpreter for Apache ZeppelinHowever it is too old.For LLAP, it need to be newer one.Properties : hive2.driver = org.apache.hive.jdbc.HiveDriverhive2.url = < can be obtained from ambari. check HiveServer2 Interactive JDBC URL >eg.hive2.url = jdbc:hive2://zzeng-hdp-3.example.com:2181,zzeng-hdp-1.example.com:2181,zzeng-hdp-2.example.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2-hive2Dependencies : org.apache.hive:hive-jdbc:2.2.0 org.apache.hadoop:hadoop-common:2.6.0 With this setting, I can access Hive LLAP now:

Integrating lucene-geo-gazetteer For Geo Parsing with Apache NiFi   lucene-geo-gazetteer is a very cool Apache Tika, Apache Lucene and Apache OpenNLP tool that builds a fast index of geo data built from a large list of all countries data. It then provides a REST API that we can easily integrate into a flow. So I have connected this to a NiFi flow for enhancing and enriching Twitter data with Geo data. Example NiFi Flow To Convert Twitter Locations Into Geo Information Downloading the Countries Data and Building the Geo Indexes Calling the Local Geo Server Example JSON Data Returned Let's Pull out the fields we want after the split Let's build a new JSON file of just the fields we like including the new geo ones.   Example JSON Processed {"msg":"RT @pauljauregui: Cybersecurity Startups Struggle - https://t.co/wADHLyUEEB #CyberSecurity #AI #IoT #IIoT #IndustrialIoT #DataSecurity #Sec…","unixtime":"1516754942404","friends_count":"4293","sentiment":"NEGATIVE","geolongitude"

There are several areas where a traditional RDBMS platform is used within an HDP environment, Ambari uses one to store the cluster configuration, Hive stores it's metastore information, Oozie stores its jobs and config and Ranger stores its policies.There are a range of DB options you can choose from for many different components, an example compatibility matrix is shown here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_support-matrices/content/ch_matrices-ambari.html#ambari_database One element that is not very well documented is how much space may be required if you're starting down the path of building a fairly large cluster. There are a number of reasons for this, the main one being that it does actually vary with how the cluster is used.However that being said I've gathered some database, cluster and time metrics from a number of production environments used by Hortonworks customers and come up with a simple formula that may at least get you a rough order of magnit

This article provides an overview of various aspects of Monitoring Hive LLAP key metrics like Hive LLAP Configurations, YARN Queue setup, YARN containers, LLAP cache hit ratio, executors, IO elevator metrics, JVM Heap usage and non-heap usage… etcExecution EngineLLAP is not an execution engine (like MapReduce or Tez). Overall execution is scheduled and monitored by an existing Hive execution engine (such as Tez) transparently over both LLAP nodes, as well as regular containers. Obviously, LLAP level of support depends on each individual execution engine (starting with Tez).MapReduce support is not planned, but other engines may be added later. Other frameworks like Pig and Spark also have the choice of using LLAP daemons.Enabling LLAP, Setting up Memory per daemon, In-Memory cache per Daemon, Number of Node(s) for running Hive LLAP daemon (num_llap_nodes_for_llap_daemons) and Number of executors per LLAP daemon in Advanced Hive Interactive-site:Cache BasicsThe daemon caches metadata fo

One task everybody faces when setting up a new Hadoop cluster, is the allocation of services. Administrators of a cluster on the other hand might ask themselves, how are my services allocated? I have discussed the visualization of HDP clusters and services more often recently and therefore decided to share my application to visualize the current and future state of a cluster. (see link to hosted app at the end of the article)What does Service Allocation mean? Planning a Hadoop cluster involves many steps and tasks that need to be considered. Almost no setup is the same (although there are some similarities). The service allocation is the part that basically tells you what services and what components will be on which node and how many nodes you have or need. This can be quiet tedious and difficult, since not all services play along well with each other, every service has different hardware/setup requirements and adding many services can get confusing. This makes it even more important

Introduction and motivation Web scraping is a technique used often times to extract data from websites for further processing. Many use cases involve data extraction from web sites as a prerequisite; the following are some examples: Data mining Online price change monitoring and price comparison Website change detection Web data integration Web scraping presents many challenges, and the complexity of the process to extract data from websites depends on the complexity of the website itself (ie. website structure, client-side frameworks, etcetera). I tend to think of web scraping as a three-stage process: Fetching and extracting web pages containing relevant data. This step is also known as web crawling. Raw data refinement, which is all about parsing, filtering, formatting and enriching the data extracted previously. More often than not, the goal of this step is to give some format to semi-structured data coming from websites. Outcome dispatching. Based on what I’ve seen in the field

If Ranger is setup to use Solr to store audits, it is necessary to configure Solr and Ranger Solr collection properly in order to keep our system stable. Here, there is no any magic number about the number of Solr nodes/shards, because with Ranger the required numbers are based on the load (generated by master components). If there are lots of ongoing jobs on your cluster, obviously it will generate more audits. (It matters more how busy is your cluster than how many machines do you have – of course, if you have more resources, then your system can handle more load)Ambari Infra Solr load test on Ranger Solr collectionIn order to get the proper (approximate) Solr settings with Ranger audits, we had to do some scale/load testing. For this purpose we also need something to visualize the results, because we need to understand what happened during the load. Here we can use AMS + Grafana. As Ambari Infra uses Solr 5.5.2, it has no Solr Metrics API in that Solr release, so we need a compone

Apache NiFi 1.0.0 has a lot of cool features, but no JSON-to-CSV converter yet.For my use case, I wanted to split one JSON file that had 5,000 records into 5,000 CSVs. You could also through a MergeContent processor in there and make one file. My data came from the excellent test data source, RandomUser. They provide a free API to pull down data, https://api.randomuser.me/?results=5000&format=pretty. I chose pretty to get formatted JSON with multiple lines which is easier to parse.This is an example of JSON returned from that service:{"results":[{"gender":"male","name":{"title":"monsieur","first":"lohan","last":"marchand"},"location":{"street":"6684 rue jean-baldassini","city":"auboranges","state":"schwyz","postcode":9591},"email":"lohan.marchand@example.com","login":{"username":"biggoose202","password":"esther","salt":"QIU1HBsr","md5":"9e60da6d4490cd6d102e8010ac98f283","sha1":"3de3ea419da1afe5c83518f8b46f157895266d17","sha256":"c6750c1a5bd18cac01c63d9e58a57d75520861733666ddb