Created a custom nifi processor, after placing nar file in /lib dir getting error on start up

[#1]

2018-07-05 17:01:05,806 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.ConvertJSONToSQL could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.ConvertJSONToSQL could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148) at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771) at org.apache.nifi.NiFi.<init>(NiFi.java:157) at org.apache.nifi.NiFi.<init>(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: java.lang.NoClassDefFoundError: org/apache/nifi/dbcp/DBCPService at org.apache.nifi.processors.standard.ConvertJSONToSQL.<clinit>(ConvertJSONToSQL.java:127) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 8 common frames omitted Caused by: java.lang.ClassNotFoundException: org.apache.nifi.dbcp.DBCPService at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 15 common frames omitted 2018-07-05 17:01:05,808 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2018-07-05 17:01:05,809 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-nar-bundles</artifactId>
        <version>1.6.0</version>
    </parent>
    <groupId>fidnifi</groupId>
    <artifactId>ExecuteSqlToAttribute</artifactId>
    <version>1.6</version>
    <packaging>nar</packaging>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <nifi.version>1.6.0</nifi.version>
    </properties>
    <repositories>
        <repository>
            <id>jcenter</id>
            <url>http://jcenter.bintray.com</url>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-api</artifactId>
            <version>${nifi.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-utils</artifactId>
            <version>${nifi.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-processor-utils</artifactId>
            <version>${nifi.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>com.jayway.jsonpath</groupId>
            <artifactId>json-path</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-mock</artifactId>
            <version>${nifi.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.10</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-standard-processors</artifactId>
            <version>${nifi.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-dbcp-service-api</artifactId>
            <version>1.6.0</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>3.8.1</version>
        </dependency>
        <dependency>
            <groupId>com.burgstaller</groupId>
            <artifactId>okhttp-digest</artifactId>
            <version>1.13</version>
            <type>jar</type>
            <scope>system</scope>
            <systemPath>${project.basedir}/src/main/resources/okhttp-digest-1.13.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>com.martiansoftware</groupId>
            <artifactId>macnificent</artifactId>
            <version>0.2.0</version>
            <scope>system</scope>
            <systemPath>${project.basedir}/src/main/resources/macnificent-0.2.0.jar</systemPath>
        </dependency>


    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.nifi</groupId>
                <artifactId>nifi-nar-maven-plugin</artifactId>
                <version>1.2.0</version>
                <extensions>true</extensions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.0</version>
            </plugin>
        </plugins>
    </build>
</project>

If I add the missing class as a jar in /lib folder error goes away, but then get another missing class error on other nifi processors. Any ideas?

[#2]

@Derek Calderon

-

If you create a custom lib directory and place your new processor jar and its dependencies in there instead of NiFi's default lib directory, do you experience the same issue(s)?

-

It is not recommended that users added any new jars or nars to NiFi's default lib directory.

Adding additional custom lib directories to NiFi is easy and also will make upgrading easier:

Simply add a new property in the nifi.properties file:

example: nifi.nar.library.directory.custom-lib1=/nifi/custom-lib1

-

Then make sure the NiFi service user has proper ownership and permissions for this directory and your custom jars/nars you place in there.

-

A restart of NiFi is needed anytime you make a configuration change to nifi.properties or add a new jar/nar to any of the lib directories.

-

Before restarting NiFi, you should delete the existing NiFi "work" directory so it is recreated cleanly after this change

-

Thank you,

Matt

-

If you found this Answer addressed your original question, please take a moment to login and click "Accept" on the answer.

[#3]

Yes the nar is in a seperate lib dir, and its still causing the issue, will try deleting the work dir see if that works. Any other ideas?

[#4]

@Derek Calderon

-

Will need a developer with more custom processor development experience then me if this did not solve issue. I just know that adding additional jars/nars to NiFi's default lib dir can introduce class loader issues with regards to dependencies shared by multiple components. If issue still exists after clearing work directory and restarting, going to need a developer to suggest possible changes to your custom code.

-

Thanks

Matt

[#5]

Earlier, Just to make sure it wasn't the code, I copied the source from ExecuteSQL 1.6 and renamed it. After deployment I still had the same errors. My suspicion is maybe the way I built the nar with its dependencies. I'm hoping that there is an error with my pom file, or like you said, deleting my work dir.

[#6]

@Matt ClarkeSo I deleted the contents of the work dir and set nifi.flowcontroller.autoResumeState=false. Still same error. I think its an issue with my pom file or dependency related at least.

[#7]

@Derek Calderon

-

Sorry to hear that. I did share this HCC link with a few devs I know if they have time to assist.

-

Thanks,

Matt