Source Code:  https://github.com/tspannhw/flume-to-nifi       Consume / Publish Kafka And Store to Files, HDFS, Hive 3.1, Kudu     Consume Kafka Flow       Merge Records And Store As AVRO or ORC   Consume Kafka, Update Records via Machine Learning Models In CDSW And Store to Kudu     Source:  Apache Kafka Topics   You enter a few parameters and start ingesting data with or without schemas.   Apache Flume had no Schema support.   Flume did not support transactions.       Sink:   Files       Storing to files in files systems, object stores, SFTP or elsewhere could not be easier.  Choose S3, Local File System, SFTP, HDFS or wherever.   Sink:   Apache Kudu / Apache Impala     Storing to Kudu/Impala (or Parquet for that manner could not be easier with Apache NiFi).     Sink:   HDFS for Apache ORC Files   When completes, the ConvertAvroToORC and PutHDFS build the Hive DDL for you!  You can build the tables automagically with Apache NiFi if you wish.   CREATE EXTERNAL TABLE IF NOT EXISTS iotsensors (sensor_id BIGINT, sensor_ts BIGINT, is_healthy STRING, response STRING, sensor_0 BIGINT, sensor_1 BIGINT, sensor_2 BIGINT, sensor_3 BIGINT, sensor_4 BIGINT, sensor_5 BIGINT, sensor_6 BIGINT, sensor_7 BIGINT, sensor_8 BIGINT, sensor_9 BIGINT, sensor_10 BIGINT, sensor_11 BIGINT) STORED AS ORC LOCATION '/tmp/iotsensors'           Sink: Kafka Publishing to Kafka is just as easy!  Push records with schema references or raw data.  AVRO or JSON, whatever makes sense for your enterprise.   Write to data easily with no coding and no changes or redeploys for schema or schema version changes.  Pick a Topic and Stream Data While Converting Types     Clean UI and REST API to Manage, Monitor, Configure and Notify on Kafka       Other Reasons to Use Apache NiFi Over Apache Flume DevOps with REST API, CLI, Python API https://community.cloudera.com/t5/Community-Articles/More-DevOps-for-HDF-Apache-NiFi-Registry-and-Friends/ta-p/248668   Schemas!   We not only work with semi-structured, structured and unstructured data.  We are schema and schema version aware for CSV, JSON, AVRO, XML, Grokked Text Files and more. https://community.cloudera.com/t5/Community-Articles/Big-Data-DevOps-Apache-NiFi-HWX-Schema-Registry-Schema/ta-p/247963 Flume Replacement Use Cases Implemented in Apache NiFi Sink/Source:   JMS https://community.cloudera.com/t5/Community-Articles/Publishing-and-Consuming-JMS-Messages-from-Tibco-Enterprise/ta-p/248157   Source:   Files/PDF/PowerPoint/Excel/Word  Sink:  Files https://community.cloudera.com/t5/Community-Articles/Parsing-Any-Document-with-Apache-NiFi-1-5-with-Apache-Tika/ta-p/247672 https://community.cloudera.com/t5/Community-Articles/Converting-PowerPoint-Presentations-into-French-from-English/ta-p/248974 https://community.cloudera.com/t5/Community-Articles/Creating-HTML-from-PDF-Excel-and-Word-Documents-using-Apache/ta-p/247968   Source:  Files/CSV  Sink:   HDFS/Hive/Apache ORC https://community.cloudera.com/t5/Community-Articles/Converting-CSV-Files-to-Apache-Hive-Tables-with-Apache-ORC/ta-p/248258   Source:  REST/Files/Simulator   Sink:  HBase, Files, HDFS.    ETL with Lookups. https://community.cloudera.com/t5/Community-Articles/ETL-With-Lookups-with-Apache-HBase-and-Apache-NiFi/ta-p/248243   Flume Replacement - Lightweight Open Source Agents   If you need to replace local Log to Kafka agents or anything to Kafka or anything to anything with routing, transformation and manipulation.   You can use Edge Flow Manager deployed MiNiFi Agents available in Java and C++ versions. References https://www.progress.com/tutorials/jdbc/ingest-salesforce-data-incrementally-into-hive-using-apache-nifi https://community.cloudera.com/t5/Community-Articles/RDBMS-to-Hive-using-NiFi-small-medium-tables/ta-p/244677 https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2018/ta-p/249363 https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2017/ta-p/247541 ... View more

Using Raspberry Pi 3B+ with Apache NiFi MiNiFi and Google Coral Accelerator and Pimoroni Inky Phat First we need to unbox our new goodies. The Inky Phat is an awesome E-Ink display with low power usage that stays displayed after shutdown! Next I added a new Google Coral Edge TPU ML Accelerator USB Coprocessor to a new Raspberry Pi 3B+. This was so easy to integrate and get up and running. Let's unbox this beautiful device (but be careful when it runs it can get really hot and there is a warning in the instructions). So I run this on top of an aluminum case and with a big fan on it. Pimoroni Inky Phat It is pretty easy to set this up and it provides a robust Python library to write to our E-Ink display. You can see an example screen here. https://github.com/pimoroni/inky Pimoroni Inky pHAT ePaper eInk Display in Red Pimoroni Inky Phat (Red) https://shop.pimoroni.com/products/inky-phat https://github.com/pimoroni/inky https://pillow.readthedocs.io/en/stable/reference/ImageDraw.html https://learn.pimoroni.com/tutorial/sandyj/getting-started-with-inky-phat Install Some Python Libraries and Debian Install for Inky PHAT and Coral pip3 install font_fredoka_one pip3 install geocoder pip3 install fswebcam sudo apt-get install fe pip3 install psutil pip3 install font_hanken_grotesk pip3 install font_intuitive wget http://storage.googleapis.com/cloud-iot-edge-pretrained-models/edgetpu_api.tar.gz These libraries are for the Inky, it needs fonts to write. The last TAR is for the Edge device and is a fast install documented well by Google. Download Apache NiFi - MiNiFi Java Agent https://nifi.apache.org/minifi/download.html Next up, the most important piece. You will need to have JDK 8 installed on your device if you are using the Java agent. You can also use the MiniFi C++ Agent but that may require building it for your OS/Platform. That has some interesting Python running abilities. Google Coral Documentation - Google Edge TPU Google Edge TPU ML accelerator coprocessor USB 3.0 Type-C socket Supports Debian Linux on host CPU ASIC designed by Google that provides high performance ML inferencing for TensorFlow Lite models https://coral.withgoogle.com/tutorials/edgetpu-retrain-classification-ondevice/ https://coral.withgoogle.com/tutorials/edgetpu-api/ http://storage.googleapis.com/cloud-iot-edge-pretrained-models/edgetpu_api_reference.zip https://coral.withgoogle.com/web-compiler/ https://coral.withgoogle.com/tutorials/edgetpu-models-intro/ https://coral.withgoogle.com/tutorials/accelerator/ https://coral.withgoogle.com/tutorials/edgetpu-api/ https://coral.withgoogle.com/models/ https://coral.withgoogle.com/tutorials/accelerator-datasheet/ Using Pretrained Tensorflow Lite Model: Inception V4 (ImageNet) Recognizes 1,000 types of objects Dataset: ImageNet Input size: 299x299 Let's run a flow: Results (Once an hour we update our E-Ink Display with Date, IP, Run Time, Label 1) Source Code https://github.com/tspannhw/nifi-minifi-coral References https://medium.freecodecamp.org/building-an-iiot-system-using-apache-nifi-mqtt-and-raspberry-pi-ce1d6ed565bc https://community.hortonworks.com/articles/85984/using-minifi-to-read-data-from-a-sense-hat-on-a-ra.html https://community.hortonworks.com/articles/107379/minifi-for-image-capture-and-ingestion-from-raspbe.html https://community.hortonworks.com/articles/107379/minifi-for-image-capture-and-ingestion-from-raspbe.html https://community.hortonworks.com/articles/32605/running-nifi-on-raspberry-pi-best-practices.html https://www.tensorflow.org/lite/convert/cmdline_examples https://www.tensorflow.org/lite/guide/get_started https://pillow.readthedocs.io/en/stable/reference/ImageDraw.html https://coral.withgoogle.com/tutorials/edgetpu-faq/ ... View more