Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
Super Guru

Integrating TensorFlow 1.6 Image Labelling with HDF 3.1 and Apache NiFi 1.5

This is community Apache NiFi custom processor that I have written with help from Simon Ball. This is after his original change, I converted the results to a different style and incorporated changes to Google's TensorFlow Label_Image in Java example.


Images Supported as Flow Files

JPG, PNG, GIF


Versions

Updated TensorFlowProcessor to TF 1.6.


Summary

I added more tests, did more clode cleanup, change the top 5 returned with cleaner naming. This processor has been tested in a few environments, please give it a try and let me know your results. The performance and stability seem quite good, hopefully enough for your use cases. You can also use the MiniFi C++ Agent's built-in C++ TensorFlow support if you wish to have fast local TensorFlow at the edge.

Video

JUNit Test for TensorFlow Processor

64813-tensorflowunittest.png

64814-tensorflowdoc.png

Here is another flow where I am send an image to two Apache MXNet Model Servers and the local TensorFlow processor.

64815-tensorflownififlow2.png

This is an example of the attributes returned by the processor. You pass in the image as a flowfile and it adds attributes without changing the contents.

64816-tensorflowoutput.png

In Apache NiFi, we can monitor how long the tasks are taking. After the first run the time of the processor goes down as we have cached the labels and the TensorFlow pre-built graph.

64817-tensorflowduration2.png

This is an example of an actual flow. I use this to grab images from Twitter, download them and run TensorFlow on them.

64818-tensorflowexample.png


Installation

  1. Download NAR here: https://github.com/tspannhw/nifi-tensorflow-processor/releases/tag/1.6
  2. Install nar file to /usr/hdf/current/nifi/lib/
  3. Create a model directory
  4. wget https://raw.githubusercontent.com/tspannhw/nifi-tensorflow-processor/master/nifi-tensorflow-processo...
  5. wget https://github.com/tspannhw/nifi-tensorflow-processor/blob/master/nifi-tensorflow-processors/src/tes...
  6. Restart Apache NiFi via Ambari

References

615 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 08:18 AM
Updated by:
 
Contributors
Top Kudoed Authors