Ingest data with NIFI to Hive LLAP and Druid Setting up Hive LLAP Setting up Druid Configuring the dimensions Setting up Superset Connection to Druid Creating a dashboard with visualisations Differences with ES/Kibana and SOLR/Banana ... View more

The standard solution Let's say you want to collect log messages from an edge cluster with NIFI, and push it to a central NIFI cluster via the Site To Site (S2S) protocol. This is exactly what NIFI is designed for, and results in a simple flow setup like this: A processor that tails the log file which sends it's flowfiles to a remote process group which is configured with the FQDN URL of the central NIFI cluster on the central NIFI cluster an INPUT port is defined and from that input port the rest of the flow is doing it's thing with the incoming flow files, like filtering, transformations and eventually sinking it into kafka, HDFS or SOLR. The NIFI S2S protocol is used for the connection between the edge NIFI cluster and the central nifi cluster. which PUSHES the flowfiles from the edge cluster to the central NIFI cluster. And now with a firewall blocking incoming connections in between This standard setup however assumes the central NIFI cluster has a public FQDN and isn't behind a firewall blocking incoming connections. But what if there is a firewall blocking incoming connections? Fear not! The flexibility of NIFI comes to the rescue once again. The solution is to move the initiation of the S2S connection from the edge NIFI to central NIFI: The remote process group in defined on the central node, which connects to a output port on the edge node as the edge NIFI node has a public FQDN (this is required!) and instead of a S2S PUSH, the data is effectively PULLED from the edge NIFI cluster to the central NIFI cluster. To be clear: this setup has the downside that the central cluster NIFI will need to know about all edge clusters. Not necessarily a big deal, just means the flow in the central NIFI cluster needs to be updated when edge clusters/nodes are added. But if you can't change the fact you have a firewall blocking incoming connections, it does the job. Example solution NIFI flow setup Screenshot of flow on Edge Node with a TailFile processor that send it's flowfiles to the output port named `logs`: Screenshot of flow on central NIFI cluster with a remote process group pointed to the FQDN of the Edge Node and a connection from the output port `logs` to the rest of the flow: The configuration of the remote process group: And the details of the `logs` connection: ... View more

To get an idea of the write performance of a Spark cluster i've created a Spark version of the standard TestDFSIO tool, which measures the I/O performance of HDFS in your cluster. Lies, damn lies and benchmarks, so the goal of this tool is providing a sanity check of your Spark setup, focusing on the HDFS writing performance, not on the compute performance. Think the tool can be improved? Feel free to submit a pull request or raise a Github issue Getting the Spark Jar Download the Spark Jar from here: https://github.com/wardbekker/benchmark/releases/download/v0.1/benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar It's build for Spark 1.6.2 / Scala 2.10.5 Or build from from source $ git clone https://github.com/wardbekker/benchmark $ cd benchmark && mvn clean package Submit args explains <file/partitions> : should ideally be equal to recommended spark.default.parallelism (cores x instances). <bytes_per_file> : should fit in memory: for example: 90000000. <write_repetitions> : no of re-writing of the test RDD to disk. benchmark will be averaged. spark-submit --class org.ward.Benchmark --master yarn --deploy-mode cluster --num-executors X --executor-cores Y --executor-memory Z target/benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar <files/partitions> <bytes_per_file> <write_repetitions> CLI Example for 12 workers with 30GB mem per node: It's important to get the amount of executors and cores right: you want to get the maximum amount of parallelism without going over the maximum capacity of the cluster. This command will write out the generated RDD 10 times, and will calculate an aggregate throughput over it. spark-submit --class org.ward.Benchmark --master yarn --deploy-mode cluster --num-executors 60 --executor-cores 3 --executor-memory 4G target/benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar 180 90000000 10 Retrieving benchmark results: You can retrieve the benchmark results by running yarn log in this way: yarn logs -applicationId <application_id> | grep 'Benchmark' for example: Benchmark: Total volume : 81000000000 Bytes Benchmark: Total write time : 74.979 s Benchmark: Aggregate Throughput : 1.08030246E9 Bytes per second So that's about 1 GB write per sec for this run. ... View more

Mindwave Neurosky The Mindwave Neurosky is a headset that allows you to record your brainwaves using EEG technology. In this article we show you how to ingest these brainwaves with NIFI Mindwave Neurosky driver installation for OSX Sierra Download and install the latest driver from http://download.neurosky.com/public/Products/MindWave%20headset/RF%20driver%20for%20Mac/MindWaveDriver5.1.pkg After the driver is installed, download and install the latest MindWave Manager from http://download.neurosky.com/public/Products/MindWave%20headset/RF%20driver%20for%20Mac/MindWave%20Manager4.0.4.zip Launch the MindWave Manager, navigate to "Pairing" section and click the "Search for MindWave", then follow the instructions to pair the headset. Install NIFI on OSX Sierra with Homebrew Install Homebrew from the terminal: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" Install NIFI (time of writing v1.1.2): brew install nifi Import NIFI Flow Template An example flow template can be downloaded using Curl: curl -O https://gist.githubusercontent.com/wardbekker/a80cbe7d12bc1866f393c5a74bf417a0/raw/9d64daa748dec352ebe8cc350e3fb34e65130ec3/mindwave_nifi_ingest_template.xml The most important processor here is the ListenTCP processor, which will listen on port 20000 and will receive the JSON payload. The flow also contains a Site 2 Site NIFI connection to a remote processgroup with the URL http://wbekkerhdf0.field.hortonworks.com:9090/nifi . You can change it to your own remote NIFI cluster. Get Ruby 'forward' script The Mindwave Thinkgear driver will create a socket where we can consume the sensor data as Json messages. To ingest it with the current vanilla version of NIFI, we need to 'forward' the messages from the thinkgear port to the NIFI ListenTCP processor port number. Upcoming versions of NIFI will have a GetTCP processor, making this Ruby script obsolete. Save this Ruby script as a file under thinkgear.rb . Run it with ruby thinkgear.rb AFTER you have connected your headset AND started ListenTCP processor on the NIFI flow. Otherwise you will run into connection errors. require 'socket' require 'json' require 'date' thinkgear_server_socket = TCPSocket.new 'localhost', 13854 nifi_server_socket = TCPSocket.new 'localhost', 20000 # trigger json output thinkgear_server_socket.puts "{\"enableRawOutput\": true, \"format\": \"Json\"}\n" while line = thinkgear_server_socket.gets # Read lines from socket hash = JSON.parse(line) hash['timestamp'] = DateTime.now.strftime('%Q') hash['user_id'] = 1 json = JSON.generate(hash) puts json nifi_server_socket.puts json end thinkgear_server_socket.close nifi_server_socket.close Start ingestion of your brainwaves Connect you headset by launching the MindWave Manager, navigate to "Pairing" section and click the "Search for MindWave", then follow the instructions to pair the headset. Start the NIFI flow, or at least the ListenTCP processor. Start the ruby script with ruby thinkgear.rb . At this point you should see JSON output from your Mindwave headset on your terminal, and new flowfiles into NIFI. Have fun with your brainwaves! ... View more