Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)
The news is out, Apache MXNet has added a Java API. So as soon as I could I got my hands on the maven repo and an example program and got to work writing a new Apache NiFi processor for it.
I have run this on standalone Apache NiFi 1.8.0 and on HDF 3.3 - Apache NiFi 1.8.0 and both work. So anyone who wants to be an alpha tester, please download it and give it a try.
Apache MXNet SSD is a good example of a pretrained deep learning model that works pretty well for general images in a use cases especially around people and cars. You can fine-tune this with some more images and runs: https://mxnet.incubator.apache.org/faq/finetune.html
The nice thing is now we can start including Apache MXNet as part of Java applications such as Kafka Streams, Apache Storm, Apache Spark, Spring Boot and other use cases using Java. I could potentially inject this into a Hive UDF (https://community.hortonworks.com/articles/39980/creating-a-hive-udf-in-java.html#comment-40026) or Pig UDF. The performance may be fast enough. We now have four Java options for Deep Learning: DL4J, H2O, Tensorflow and Apache MXNet. Unfortunately, both TensorFlow and MXNet Java APIs are not quite production ready.
I may do some further research on running MXNet as a Hive UDF, it would be cool to have in a query.
For those who don't want to setup a development environment with JDK 8+, Maven 3.3+ and git, you can download a pre-built nar file here: https://github.com/tspannhw/nifi-mxnetinference-processor/releases/tag/v1.0.
As part of the recent release of HDF 3.3, I have upgraded my OpenStack Centos 7 cluster.
Important Caveats
Notice, the Java API is in preview and so is this processor. Do not use this in production! This is in development and I am the only one working on it. The Java API from Apache MXNet is in flux and will be changing. See the POM as it is tied to the OSX/Mac version of the library. You will need to change that. You will need to download the pre-built MXNet model and place it in a directory accessible to Apache NiFi server/cluster. I am still cleaning up the rectangle code for identifying objects in the pictures.
As you will notice, my rectangle drawing is a bit off. I need to work on that.
Once you drop your built nar file and models in the nifi/lib directory and restart Apache NiFi, you can add it to your canvas.
We need to feed it some images. You can use my web cam processor, an image URL feed or local files.
To grab images from an HTTPS site, you need an SSL Context Service like this StandardSSLContextService below. You will need to point to the cacerts used by the JRE/JDK running your Apache NiFi node. The default password in Java is changeme. Hopefully you have changed it.
To configure my new processor, just put in the full path to the model directory and then "/resnet50_ssd_model" as that is the prefix for the model.
Our example flow with new processor being fed by traffic cameras, webcams, local files and local webcam.
Some output of our flow:
Our top 5 probabilities and labels
Example Data:
{
"ymin_1" : "456.01",
"ymin_5" : "159.29",
"ymin_4" : "235.83",
"ymin_3" : "206.64",
"ymin_2" : "383.84",
"label_5" : "person",
"xmax_5" : "121.14",
"label_4" : "bicycle",
"xmax_4" : "137.89",
"label_3" : "dog",
"xmax_3" : "179.14",
"ymax_1" : "150.66",
"ymax_2" : "418.95",
"ymax_3" : "476.79",
"label_2" : "bicycle",
"label_1" : "car",
"probability_4" : "0.22",
"probability_5" : "0.13",
"probability_2" : "0.90",
"xmin_5" : "88.93",
"probability_3" : "0.82",
"ymax_4" : "413.43",
"probability_1" : "1.00",
"ymax_5" : "190.04",
"xmax_2" : "149.96",
"xmax_1" : "72.03",
"xmin_3" : "83.82",
"xmin_4" : "93.05",
"xmin_1" : "312.21",
"xmin_2" : "155.96"
}Resources:
- https://medium.com/apache-mxnet/introducing-java-apis-for-deep-learning-inference-with-apache-mxnet-...
- https://github.com/apache/incubator-mxnet/tree/java-api/scala-package/examples/src/main/java/org/apa...
- https://mxnet.incubator.apache.org/install/java_setup.html
Source:
https://github.com/tspannhw/nifi-mxnetinference-processor
Video walk-through: https://www.youtube.com/watch?v=Q4dSGPvqXSA&t=196s&list=PL-7XqvSmQqfTSihuoIP_ZAnN7mFIHkZ_e&index=17
Download the artifacts listed: https://github.com/apache/incubator-mxnet/tree/java-api/scala-package/examples/src/main/java/org/apa...
Maven POM (I used Java 8 and Maven 3.3.9)
<?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>com.dataflowdeveloper.mxnet</groupId> <artifactId>inference</artifactId> <version>1.0</version> </parent> <artifactId>nifi-mxnetinference-processors</artifactId> <packaging>jar</packaging> <dependencies> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-api</artifactId> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-utils</artifactId> <version>1.8.0</version> </dependency> <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-mock</artifactId> <version>1.8.0</version> <scope>test</scope> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.mxnet</groupId> <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId> <version>1.3.1-SNAPSHOT</version> </dependency> </dependencies> </project>
I have moved from Eclipse to IntelliJ from my builds. I am looking at Apache Netbeans as well.
