Member since
06-01-2016
5
Posts
0
Kudos Received
0
Solutions
06-01-2016
06:54 PM
I have read some documentation and this looks correct. Tf I wanted to create a custom processor that leverages the DBCPConnectionPool Controller Service, I believe these are the correct dependences to include: <!-- NiFi DBCP Service -->
<!-- DBCP API -->
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-api</artifactId>
<version>${apache.nifi.version}</version>
</dependency>
<!-- DBCP NAR -->
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service-nar</artifactId>
<version>${apache.nifi.version}</version>
<type>nar</type>
</dependency>
... View more
06-01-2016
04:49 PM
After reviewing the example, it looks like modifying the dependencies should fix the problem <dependencies>
<!-- Testing/Mocking -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<version>${nifi.version}</version>
</dependency>
<!-- NiFi Dependencies -->
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-distributed-cache-client-service-api</artifactId>
<version>${nifi.version}</version>
</dependency>
<!-- NiFi DBCP Service -->
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-dbcp-service</artifactId>
<version>0.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
<!-- <version>${nifi.version}</version> -->
</dependency>
<!-- Helper Services -->
<dependency>
<groupId>x-y</groupId>
<artifactId>y-z-common</artifactId>
<version>${y.version}</version>
</dependency>
<dependency>
<groupId>x-y</groupId>
<artifactId>y-core</artifactId>
<version>${y.version}</version>
</dependency>
<!-- Custom Processor Dependencies -->
<dependency>
<groupId>net.sourceforge.javacsv</groupId>
<artifactId>javacsv</artifactId>
</dependency>
</dependencies>
I assumed that you would just import the jars, but wiring dependent nars sounds like the best way to go. Could you take a look at the dependencies section of my pom and give pointers on how to convert the nifi-dbcp-service dependency into a nar dependency?
... View more
06-01-2016
03:09 PM
I am currently using HDF 1.1.1.0-12 and the version of NiFi bundled with HDF. NiFi runs into a classpath/dependency problem when I try and create custom processors referencing controller/client services in Apache NiFi jars from Maven Central (I have tried versions 0.4.0, 0.4.1, 0.5.0, 0.5.1, and 0.6.1). Here is the specific warning/error message that comes up in the logs: 2016-05-31 11:40:00,285 WARN [main] org.apache.nifi.nar.ExtensionManager Attempt was made to load org.apache.nifi.dbcp.DBCPConnectionPool from org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-dbcp-service-nar-1.1.1.0-12.nar-unpacked] but that class name is already loaded/registered from org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/xxx-yyy-zzz-0.3.0-SNAPSHOT.nar-unpacked]. This may cause unpredictable behavior. Order of NARs is not guaranteed.
The .nar that contains my custom processor and the Apache NiFi jar is packaged as xxx-yyy-zzz.nar and this conflicts with the internal .jar that HDF NiFi uses. As a result, my custom processor can be implemented, but does not work with the DBCPConnectionPool Controller Service that I need to query a Postgres table. When I do try and choose a new Controller Service to use, I get a message saying: "HBase Client Service" validated against [uuid] is invalid because Invalid Controller Service [uuid] is not a valid Controller Service identifier or does not reference the correct type of Controller Service. The uuid in brackets are the same values. Are there Apache NiFi jars that play nicely with HDF NiFi 1.1.1.0-12? Do HDF NiFi jars exist?
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)