Build Custom Nifi Processor

[#1]

1Nifi Custom Processor Overview

Apache nifi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Its very simple to use product using which you can build "data flow" very easily. As of version 3.0 it has 90 prebuilt processors but then again you can extend them by adding your own custom processors.

In this article I am going to talk about how you can build a custom nifi processor in your local machine and then move the final finished processor which is a nar file to nifi.

This article is based on a video from youtube and here is the link for that

https://www.youtube.com/watch?v=3ldmNFlelhw

2Steps to build Customer processor

Here are the steps involved to build the custom processor for nifi. I used my mac to build this processor

2.1Required Software

Two software that you would need in you location machine are

• 1.maven
• 2.Java

Here is how you can quickly check if you have them installed

mvn -version

java -version

Here are the results from my machine

$ mvn -version

Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T11:29:23-06:00)

Maven home: /usr/local/Cellar/maven/3.2.5/libexec

Java version: 1.8.0_65, vendor: Oracle Corporation

Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "mac os x", version: "10.10.4", arch: "x86_64", family: "mac"

$ java -version

java version "1.8.0_65"

Java(TM) SE Runtime Environment (build 1.8.0_65-b17)

Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)

2.2Create a directory where you want to build the processer

I created the directory under following location

cd <Home Dir>/Documents/nifi/ChakraProcessor

mkdir ChakraProcessor

2.3Create the nifi processor with default value

Get to that new directory you just created and use mvn command to build the required java files

cd <Home Dir>/Documents/nifi/ChakraProcessor

mvn archetype:generate

You will be asked for bunch of parameters. I choose following parameters which are highlighted in bold

Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): 690: nifi

Choose archetype:

1: remote -> org.apache.nifi:nifi-processor-bundle-archetype (-)

2: remote -> org.apache.nifi:nifi-service-bundle-archetype (-)

Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1

Choose org.apache.nifi:nifi-processor-bundle-archetype version:

1: 0.0.2-incubating

2: 0.1.0-incubating

3: 0.2.0-incubating

4: 0.2.1

5: 0.3.0

Choose a number: 5: 4

Downloading: https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-processor-bundle-archetype/0.2.1/nifi-proc...

Downloaded: https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-processor-bundle-archetype/0.2.1/nifi-proc... (12 KB at 8.0 KB/sec)

Downloading: https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-processor-bundle-archetype/0.2.1/nifi-proc...

Downloaded: https://repo.maven.apache.org/maven2/org/apache/nifi/nifi-processor-bundle-archetype/0.2.1/nifi-proc... (2 KB at 9.4 KB/sec)

Define value for property 'groupId': : hwx

Define value for property 'artifactId': : HWX

Define value for property 'version': 1.0-SNAPSHOT: : 1.0

Define value for property 'artifactBaseName': : demo

Define value for property 'package': hwx.processors.demo: :

[INFO] Using property: nifiVersion = 0.1.0-incubating-SNAPSHOT

Confirm properties configuration:

groupId: hwx

artifactId: HWX

version: 1.0

artifactBaseName: demo

package: hwx.processors.demo

nifiVersion: 0.1.0-incubating-SNAPSHOT

Y: : Y

[INFO] ----------------------------------------------------------------------------

[INFO] Using following parameters for creating project from Archetype: nifi-processor-bundle-archetype:0.2.1

[INFO] ----------------------------------------------------------------------------

2.4Modify the processor

Above command will result in a MyProcessor.java file which is where you will put your code for your custom processor

Open MyProcessor.java under following location

<Home Dir>/Documents/nifi/ChakraProcessor/HWX/nifi-demo-processors/src/main/java/hwx/processors

Add following lines at the end after //TODO implement section

// TODO implement

System.out.println("This is a custom processor that will receive flow file");

session.transfer(flowFile,MY_RELATIONSHIP);

2.5Change POM

There is change that you need to make to the POM file before you can create the package.

Remove the -Snapshot from pom.xml file under following location

<Home Dir>/Documents/nifi/ChakraProcessor/HWX

2.6Create nar file for your processor

cd <Home Dir>/Documents/nifi/ChakraProcessor/HWX

mvn install

Once maven install is done you will have the nar file at the target directory with name nifi-demo-nar-1.0.nar

cd <Home Dir>/Documents/nifi/ChakraProcessor/HWX/nifi-demo-nar/target

$ ls

classes maven-archiver maven-shared-archive-resources nifi-demo-nar-1.0.nar

2.7Copy the nar file to Nifi

Copy the nar file to the bin directory of where nifi is installed

Here is a sample command to copy the file --

scp nifi-demo-nar-1.0.nar root@172.16.149.157:/opt/nifi-1.0.0.0-7/bin/

Once the nar file is copied youi need to restart nifi

Once restarted you should be able to add the custom processor that you built which will show up with the name "MyProcessor"

2.8Build Nifi data flow

You can build a new data flow using this customer processor

GenerateFlowFile --> MyProcessor --> LogAttribute

For "MyProcessor" you can enter some random value under the property section to make it valid.

[#2]

There is incorrect example

scp nifi-demo-nar-1.0.nar root@172.16.149.157:/opt/nifi-1.0.0.0-7/bin/

It seems that you mean lib directory and example should look like

scp nifi-demo-nar-1.0.nar root@172.16.149.157:/opt/nifi-1.0.0.0-7/lib/

[#3]

Section 2.7. You probably meant /lib instead of /bin. That's where nar files are deployed.

[#4]

For the beginners, like myself: If you added a new processor, or change the processor name, you will need to add or change the name in the .Processor file in <Home Dir>/Documents/nifi/ChakraProcessor/HWX/nifi-demo-processors/src/main/resources/META-INF/services.

If you don't do this, the processor will not be loaded.