Executing TensorFlow Classifications from Apache NiFi Using Apache Spark 2.3 and Apache Livy
Technology: Apache Spark 2.3 + Apache Livy + Apache NiFi 1.5 + TensorFlow + Python
Python Code: https://github.com/tspannhw/DWS-DeepLearning-CrashCourse/blob/master/tensorflowsparknifi.py
TIP: In this version of Apache NiFi, you need to use double quotes (") instead of single quotes (') in your Python code.
Python Code for NiFi ExecuteSparkInteractive
See Github
Simple Apache NiFi Flow To Execute TensorFlow Python Applications via Apache Livy
I am just using Apache Livy as the transport call from Apache NiFi to Apache Spark. My Apache Spark 2.3 cluster is not doing any Spark specific processing. PySpark is just running a vanilla TensorFlow python application in this version. We could also call TensorFlow on Spark code in this way. My goal was to run TensorFlow on my Spark cluster trigger from Apache NiFi and get back results.
Results Returned in Success From ExecuteSparkInteractive Call
{
"text/plain" : "273\tracer, race car, racing car\t37.4601334333%\n\n274\tsports car, sport car\t25.3520905972%\n\n267\tcab, hack, taxi, taxicab\t11.1182622612%\n\n268\tconvertible\t9.85431224108%\n\n271\tminivan\t3.22951599956%"
}Apache Livy UI Showing Results of Runs
This is the ExecuteSparkInteractive Processor. We can put the code in the Code property or pass it in.
Let's Configure a PySpark Apache Livy Controller
LogSearch
There is a technical preview of LogSearch which is great for finding issues in HDF components or HDP components. This is easier then searching logs. Though I can easily write NiFi code to search logs as well.
References:
