Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Solved Go to solution
Highlighted

[RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Explorer

Hi all,

I need to extend logback usage. I want to add new jars like :

EXTRA_CLASSPATH=logstash-logback-encoder-4.7.jar

It is possible to add to bootstrap.conf or nifi.sh at startup of nifi to use this jars ?

thanks.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Any jars directly in the NiFI lib directory are available on the system classpath, and are thus available to all NARs, and the class loading is done parent first, so if foo.jar is in lib and foo.jar is also bundled in a NAR, it will find the one from lib first.

Generally it is not recommended to put additional JARs into the NiFi lib directory since it can impact every NAR, and the point of the NARs is for each of them to have isolated class loading. However, in your case if the JAR is just an additional logback JAR to reference classes in logback.xml then you should be ok to put in the lib directory, just be aware that it is now on the classpath for all NARs as well.

View solution in original post

6 REPLIES 6
Highlighted

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Any jars directly in the NiFI lib directory are available on the system classpath, and are thus available to all NARs, and the class loading is done parent first, so if foo.jar is in lib and foo.jar is also bundled in a NAR, it will find the one from lib first.

Generally it is not recommended to put additional JARs into the NiFi lib directory since it can impact every NAR, and the point of the NARs is for each of them to have isolated class loading. However, in your case if the JAR is just an additional logback JAR to reference classes in logback.xml then you should be ok to put in the lib directory, just be aware that it is now on the classpath for all NARs as well.

View solution in original post

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Explorer

@Bryan Bende

As your explain, I've put the jars to lib but NIFI didn't start :

-rw-r--r-- 1 root root  105112 Jul 17  2006 servlet-api-2.5.jar
-rw-r--r-- 1 root root   46968 Jul 17  2015 jackson-annotations-2.6.0.jar
-rw-r--r-- 1 root root  258876 Jan 19  2016 jackson-core-2.6.5.jar
-rw-r--r-- 1 root root 1171380 Jan 19  2016 jackson-databind-2.6.5.jar
-rw-r--r-- 1 root root    3493 Feb 24  2016 animal-sniffer-annotations-1.15.jar
-rw-r--r-- 1 root root  452348 May  2  2016 logstash-logback-encoder-4.7.jar

2017-02-02 10:41:25,698 INFO [main] org.eclipse.jetty.server.Server jetty-9.3.9.v20160517
2017-02-02 10:41:26,157 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [j2ee_web_services_1_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,158 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_1_2.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-app_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-fragment_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-common_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_6.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_1_3.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_client_1_3.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-app_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-fragment_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-common_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,161 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_7.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,161 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,161 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_client_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,211 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@6765b654{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2017-02-02 10:41:26,222 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.NoSuchMethodError: javax.servlet.ServletContext.getJspConfigDescriptor()Ljavax/servlet/descriptor/JspConfigDescriptor;
java.lang.NoSuchMethodError: javax.servlet.ServletContext.getJspConfigDescriptor()Ljavax/servlet/descriptor/JspConfigDescriptor;
        at org.apache.jasper.servlet.TldScanner.scanJspConfig(TldScanner.java:158) ~[apache-jsp-8.0.33.jar:2.3]
        at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104) ~[apache-jsp-8.0.33.jar:2.3]
        at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:103) ~[apache-jsp-8.0.33.jar:2.3]
        at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:140) ~[jetty-plus-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:63) ~[jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:330) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.start(Server.java:411) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.doStart(Server.java:378) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675) ~[nifi-jetty-1.1.0.jar:1.1.0]
        at org.apache.nifi.NiFi.<init>(NiFi.java:156) ~[nifi-runtime-1.1.0.jar:1.1.0]
        at org.apache.nifi.NiFi.main(NiFi.java:262) ~[nifi-runtime-1.1.0.jar:1.1.0]

There are some issues or incompatible with JAR ?

thanks

Highlighted

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Ok in your original question I thought you wanted to add only logstash-logback-encoder-4.7.jar which I think on its own would be fairly safe to be in the lib directory since most other NARs would not be using this JAR, and other logging JARs are already in lib.

Putting servler-api and jackson JARs in the lib will likely cause a problem. A lot of NARs use jackson and could be using different versions than what you put in the lib, and NiFi itself runs Jetty which is using the servlet-api and probably is using a different version that the one you put in the lib which I think is causing the problem in this case.

Highlighted

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Explorer

@Bryan Bende : after somes tests, i've removed Servlet-API and Jackson.

Now i've another message : it seems that NIFI can instantiate LogstashAppender

15:48:28,049 |-ERROR in ch.qos.logback.core.joran.action.AppenderAction - Could not create an Appender of type [net.logstash.logback.appender.LogstashTcpSocketAppender]. ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender
        at ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassNameAndParameter(OptionHelper.java:73)
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassName(OptionHelper.java:48)
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassName(OptionHelper.java:35)
        at      at ch.qos.logback.core.joran.action.AppenderAction.begin(AppenderAction.java:54)
        at      at ch.qos.logback.core.joran.spi.Interpreter.callBeginAction(Interpreter.java:275)
        at      at ch.qos.logback.core.joran.spi.Interpreter.startElement(Interpreter.java:147)
        at      at ch.qos.logback.core.joran.spi.Interpreter.startElement(Interpreter.java:129)
        at      at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:50)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
        at      at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
        at      at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
        at      at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
        at      at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
        at      at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
        at      at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
        at      at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
        at      at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
        at      at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:123)
        at      at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:205)
Caused by: java.lang.ClassNotFoundException: net.logstash.logback.appender.LogstashTcpSocketAppender
        at      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassNameAndParameter(OptionHelper.java:60)
        at      ... 21 common frames omitted
15:48:28,049 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@44:94 - ActionException in Action for tag [appender] ch.qos.logback.core.joran.spi.ActionException: ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender
        at ch.qos.logback.core.joran.spi.ActionException: ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender

Here where i've put the jar

[root@devnode016 lib]# ls /var/opt/hosting/nifi-1.1.0/lib/logstash-logback-encoder-4.7.jar
/var/opt/hosting/nifi-1.1.0/lib/logstash-logback-encoder-4.7.jar

Highlighted

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Not really sure about that one, can you provide the part of logback.xml where you declared the new appender?

Highlighted

Re: [RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Explorer

Hi all,

I'll found that some wrong. So if you need use logstash-encoder with NIFI.

Make these modifications :

Links jackson jar from bootstrap to lib

lrwxrwxrwx 1 root root       71 Feb  3 15:04 jackson-annotations-2.6.0.jar -> lib/bootstrap/jackson-annotations-2.6.0.jar
lrwxrwxrwx 1 root root       64 Feb  3 15:04 jackson-core-2.6.1.jar -> lib/bootstrap/jackson-core-2.6.1.jar
lrwxrwxrwx 1 root root       68 Feb  3 15:04 jackson-databind-2.6.1.jar -> lib/bootstrap/jackson-databind-2.6.1.jar

Add logstash-logback-encoder-4.7.jar to lib

and configure logback.xml with Appender.

Don't have an account?
Coming from Hortonworks? Activate your account here