HDF NiFi Version and Apache NiFi Version Conflict

[#1]

I am currently using HDF 1.1.1.0-12 and the version of NiFi bundled with HDF. NiFi runs into a classpath/dependency problem when I try and create custom processors referencing controller/client services in Apache NiFi jars from Maven Central (I have tried versions 0.4.0, 0.4.1, 0.5.0, 0.5.1, and 0.6.1). Here is the specific warning/error message that comes up in the logs:

2016-05-31 11:40:00,285 WARN [main] org.apache.nifi.nar.ExtensionManager Attempt was made to load org.apache.nifi.dbcp.DBCPConnectionPool from org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-dbcp-service-nar-1.1.1.0-12.nar-unpacked] but that class name is already loaded/registered from org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/xxx-yyy-zzz-0.3.0-SNAPSHOT.nar-unpacked].  This may cause unpredictable behavior.  Order of NARs is not guaranteed.

The .nar that contains my custom processor and the Apache NiFi jar is packaged as xxx-yyy-zzz.nar and this conflicts with the internal .jar that HDF NiFi uses. As a result, my custom processor can be implemented, but does not work with the DBCPConnectionPool Controller Service that I need to query a Postgres table. When I do try and choose a new Controller Service to use, I get a message saying:

"HBase Client Service" validated against [uuid] is invalid because Invalid Controller Service [uuid] is not a valid Controller Service identifier or does not reference the correct type of Controller Service.

The uuid in brackets are the same values.

Are there Apache NiFi jars that play nicely with HDF NiFi 1.1.1.0-12? Do HDF NiFi jars exist?

[#2 · Accepted solution]

What looks likely to be happening is that your nar is bundling things you did not intend to include or should not include. Specifically you will not want to include the actual implementations of services you're depending on. You can have a nar depend on another nar and NiFi will automatically not actually pull that dependent nar into your nar but rather will wire it up at startup as part of its classpath handling. However, if you include the implementation of other services they could conflict as I suspect is the case here. Could you share/show details of how your project it structure or provide a pom and we could probably give you more specific pointers. There are several examples of nars in the apache nifi source too that show how to depend on services without pulling in their implementations. This is a good example https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/p... It pulls in the nar for the service definitions so that this nar can depend on those things but not actually include them in it's own package or create conflicts.

[#3]

After reviewing the example, it looks like modifying the dependencies should fix the problem

<dependencies>
	<!-- Testing/Mocking -->
	<dependency>
		<groupId>junit</groupId>
		<artifactId>junit</artifactId>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>org.slf4j</groupId>
		<artifactId>slf4j-log4j12</artifactId>
		<scope>test</scope>
	</dependency>
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-mock</artifactId>
		<version>${nifi.version}</version>
	</dependency>
	<!-- NiFi Dependencies -->
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-api</artifactId>
	</dependency>
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-utils</artifactId>
	</dependency>
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-processor-utils</artifactId>
	</dependency>
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-distributed-cache-client-service-api</artifactId>
		<version>${nifi.version}</version>
	</dependency>
	<!-- NiFi DBCP Service -->
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-dbcp-service</artifactId>
		<version>0.4.0</version>
	</dependency>
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-utils</artifactId>
		<!-- 		<version>${nifi.version}</version> -->
	</dependency>
	<!-- Helper Services -->
	<dependency>
		<groupId>x-y</groupId>
		<artifactId>y-z-common</artifactId>
		<version>${y.version}</version>
	</dependency>
	<dependency>
		<groupId>x-y</groupId>
		<artifactId>y-core</artifactId>
		<version>${y.version}</version>
	</dependency>
	<!-- Custom Processor Dependencies -->
	<dependency>
		<groupId>net.sourceforge.javacsv</groupId>
		<artifactId>javacsv</artifactId>
	</dependency>
</dependencies>

I assumed that you would just import the jars, but wiring dependent nars sounds like the best way to go. Could you take a look at the dependencies section of my pom and give pointers on how to convert the nifi-dbcp-service dependency into a nar dependency?

[#4]

I have read some documentation and this looks correct. Tf I wanted to create a custom processor that leverages the DBCPConnectionPool Controller Service, I believe these are the correct dependences to include:

	<!-- NiFi DBCP Service -->
	<!-- DBCP API -->
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-dbcp-service-api</artifactId>
		<version>${apache.nifi.version}</version>
	</dependency>
	<!-- DBCP NAR -->
	<dependency>
		<groupId>org.apache.nifi</groupId>
		<artifactId>nifi-dbcp-service-nar</artifactId>
		<version>${apache.nifi.version}</version>
		<type>nar</type>
	</dependency>