Is there any command to set classpath in Apache Nifi

[#1]

Can we set classpath using "java -classpath " command in Nifi like we do in other CLI hadoop component. If not than Is it possible to set this in bootstrap.conf file or Nifi.sh script. I have seen a way in this post https://issues.apache.org/jira/browse/NIFI-416

however not sure this is possible or not.

[#2 · Accepted solution]

Can you give some background on why you want to set the classpath?

NiFi has a very specific class-loader isolation model and generally you don't add JARs to NiFi's classpath like you do with other projects.

[#3]

Is that mean NIFI works independently and can't be integrated with other maven projects?

In my use case i have created a NAR project which has a custom processor class and from here i am trying to call some functions which are available in other existing modules. These modules have a lot of third party jars and config property files. i am unable to find where should i keep these external modules/jars in NIFI deployment setup on server so that my each nifi instance of cluster can access it. I have tried putting all jars in ./lib and ./conf folder with other nar and conf files but no luck.

[#4]

If you created a NAR project you should have a project structure similar to what is shown here:

https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExten...

Your processors module would have Maven dependencies on all of the modules you need to call from your processor code and when you build your NAR they would all get packaged into your NAR.

If you need config files on the classpath, you would likely put them in src/main/resources of your processors module. If the config files need to be editable, you could have a property in your processor that specifies a directory of where to load config from and then have code in your processor to load them in,

In the end you would generally put only the NAR into NiFi's lib directory.

[#5]

Hi Bryan, I have similar Nifi project structure as shown in link above. I have already tried the way you suggested but didn't make any progress. please check attached screenshot this is how my program flow goes. Likewise i have 4 to 5 modules which are dependent on each other. i have added dependency of each module in my NAR pom.xml and i am not having any compilation issue. i am not sure how Nifi will collect all dependent jars along with third party and cluster specific jars and bundle in a NAR.

[#6]

Can you share your NAR and processors project, or at least the pom.xml from each project? It is going to be hard to help without that.

You should have something like:

custom-nar pom.xml with dependency on custom-processor jar

custom-processor pom.xml with dependency on module1 jar

At that point, whatever module1 is dependent on would also get brought in.

[#7]

nifi-nar-pom.xml nifi-processor-pom.xml

@Bryan Bende

Attached are nar-pom and custom-processor-pom xmls.

My processor is calling some method from other modules and these modules are part of a complete product. so i have to deploy all the common-lib,conf-files and module libs inside nifi setup so that it is accessible for all.

I am still not able to understand while building NAR project how all this will get wrapped inside a single NAR.

[#8]

Ok thanks for providing that... When the NAR module gets built it uses a Maven plugin for packaging a NAR, this based on the packaging element in the pom.xml saying "nar". That plugin takes all of the dependencies of the NAR module, and all of their transitive dependencies, and packages them into a single archive (the NAR). After running a build, if you take the NAR file from the target directory and unzip it, you should see a directory like "META-INF/bundled-dependencies/" and that will show you all the JARs that are included in the NAR.

In your example when the NAR module is built it will first see the dependency on nifi-demo-processors, so it will include that JAR, then it will go to through the dependencies of nifi-demo-processors and include all those JARs, and also their transitive dependencies. So in your screenshot, if you want everything in common-lib and lib to be accessible to your processor, then all of those JARs need to be listed as a dependency in the processors pom, or need to be transitive dependencies of something listed in the processors pom.

[#9]

Thanks for your quick response. Although i have already added all transitive dependencies still i would again have a look. i just had one question where does this external config files will go by default when i prepare a NAR. i unzipped my NAR, it have all the jars but no sign of config files.

[#10]

There isn't a specific place for config files in a NAR, a NAR is only bundle of JARs, so you would have to put all the config files in your processors project in src/main/resources which would bundle them into the processors JAR so they are on the classpath.