Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Solved Go to solution

[RESOLVED] NIFI : ADD EXTRACLASSPATH at startup

Labels:
avatar
author-rank maykiwogno
Rising Star

Created ‎02-01-2017 03:14 PM

Hi all,

I need to extend logback usage. I want to add new jars like :

EXTRA_CLASSPATH=logstash-logback-encoder-4.7.jar

It is possible to add to bootstrap.conf or nifi.sh at startup of nifi to use this jars ?

thanks.

3,811 Views
1 ACCEPTED SOLUTION

avatar
author-rank bbende author-rank
Master Guru

Created ‎02-01-2017 06:50 PM

Any jars directly in the NiFI lib directory are available on the system classpath, and are thus available to all NARs, and the class loading is done parent first, so if foo.jar is in lib and foo.jar is also bundled in a NAR, it will find the one from lib first.

Generally it is not recommended to put additional JARs into the NiFi lib directory since it can impact every NAR, and the point of the NARs is for each of them to have isolated class loading. However, in your case if the JAR is just an additional logback JAR to reference classes in logback.xml then you should be ok to put in the lib directory, just be aware that it is now on the classpath for all NARs as well.

View solution in original post

3,255 Views
6 REPLIES 6

avatar
author-rank bbende author-rank
Master Guru

Created ‎02-01-2017 06:50 PM

Any jars directly in the NiFI lib directory are available on the system classpath, and are thus available to all NARs, and the class loading is done parent first, so if foo.jar is in lib and foo.jar is also bundled in a NAR, it will find the one from lib first.

Generally it is not recommended to put additional JARs into the NiFi lib directory since it can impact every NAR, and the point of the NARs is for each of them to have isolated class loading. However, in your case if the JAR is just an additional logback JAR to reference classes in logback.xml then you should be ok to put in the lib directory, just be aware that it is now on the classpath for all NARs as well.

3,256 Views

avatar
author-rank maykiwogno
Rising Star

Created ‎02-02-2017 09:52 AM

@Bryan Bende

As your explain, I've put the jars to lib but NIFI didn't start :

-rw-r--r-- 1 root root  105112 Jul 17  2006 servlet-api-2.5.jar
-rw-r--r-- 1 root root   46968 Jul 17  2015 jackson-annotations-2.6.0.jar
-rw-r--r-- 1 root root  258876 Jan 19  2016 jackson-core-2.6.5.jar
-rw-r--r-- 1 root root 1171380 Jan 19  2016 jackson-databind-2.6.5.jar
-rw-r--r-- 1 root root    3493 Feb 24  2016 animal-sniffer-annotations-1.15.jar
-rw-r--r-- 1 root root  452348 May  2  2016 logstash-logback-encoder-4.7.jar


2017-02-02 10:41:25,698 INFO [main] org.eclipse.jetty.server.Server jetty-9.3.9.v20160517
2017-02-02 10:41:26,157 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [j2ee_web_services_1_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,158 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_1_2.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-app_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-fragment_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-common_3_0.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,159 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_6.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_1_3.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_client_1_3.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-app_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-fragment_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,160 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [web-common_3_1.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,161 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_7.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,161 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,161 WARN [main] o.a.t.util.descriptor.DigesterFactory The XML schema [javaee_web_services_client_1_4.xsd] could not be found. This is very likely to break XML validation if XML validation is enabled.
2017-02-02 10:41:26,211 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@6765b654{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2017-02-02 10:41:26,222 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.NoSuchMethodError: javax.servlet.ServletContext.getJspConfigDescriptor()Ljavax/servlet/descriptor/JspConfigDescriptor;
java.lang.NoSuchMethodError: javax.servlet.ServletContext.getJspConfigDescriptor()Ljavax/servlet/descriptor/JspConfigDescriptor;
        at org.apache.jasper.servlet.TldScanner.scanJspConfig(TldScanner.java:158) ~[apache-jsp-8.0.33.jar:2.3]
        at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104) ~[apache-jsp-8.0.33.jar:2.3]
        at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:103) ~[apache-jsp-8.0.33.jar:2.3]
        at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:140) ~[jetty-plus-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:63) ~[jetty-annotations-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:330) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.start(Server.java:411) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.doStart(Server.java:378) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675) ~[nifi-jetty-1.1.0.jar:1.1.0]
        at org.apache.nifi.NiFi.<init>(NiFi.java:156) ~[nifi-runtime-1.1.0.jar:1.1.0]
        at org.apache.nifi.NiFi.main(NiFi.java:262) ~[nifi-runtime-1.1.0.jar:1.1.0]

There are some issues or incompatible with JAR ?

thanks

3,255 Views
0 Kudos

avatar
author-rank bbende author-rank
Master Guru

Created ‎02-02-2017 03:11 PM

Ok in your original question I thought you wanted to add only logstash-logback-encoder-4.7.jar which I think on its own would be fairly safe to be in the lib directory since most other NARs would not be using this JAR, and other logging JARs are already in lib.

Putting servler-api and jackson JARs in the lib will likely cause a problem. A lot of NARs use jackson and could be using different versions than what you put in the lib, and NiFi itself runs Jetty which is using the servlet-api and probably is using a different version that the one you put in the lib which I think is causing the problem in this case.

3,255 Views
0 Kudos

avatar
author-rank maykiwogno
Rising Star

Created ‎02-02-2017 03:20 PM

@Bryan Bende : after somes tests, i've removed Servlet-API and Jackson.

Now i've another message : it seems that NIFI can instantiate LogstashAppender 

15:48:28,049 |-ERROR in ch.qos.logback.core.joran.action.AppenderAction - Could not create an Appender of type [net.logstash.logback.appender.LogstashTcpSocketAppender]. ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender
        at ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassNameAndParameter(OptionHelper.java:73)
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassName(OptionHelper.java:48)
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassName(OptionHelper.java:35)
        at      at ch.qos.logback.core.joran.action.AppenderAction.begin(AppenderAction.java:54)
        at      at ch.qos.logback.core.joran.spi.Interpreter.callBeginAction(Interpreter.java:275)
        at      at ch.qos.logback.core.joran.spi.Interpreter.startElement(Interpreter.java:147)
        at      at ch.qos.logback.core.joran.spi.Interpreter.startElement(Interpreter.java:129)
        at      at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:50)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
        at      at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
        at      at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:77)
        at      at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152)
        at      at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
        at      at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
        at      at org.slf4j.LoggerFactory.bind(LoggerFactory.java:141)
        at      at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
        at      at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
        at      at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
        at      at org.apache.nifi.bootstrap.RunNiFi.<init>(RunNiFi.java:123)
        at      at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:205)
Caused by: java.lang.ClassNotFoundException: net.logstash.logback.appender.LogstashTcpSocketAppender
        at      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at      at ch.qos.logback.core.util.OptionHelper.instantiateByClassNameAndParameter(OptionHelper.java:60)
        at      ... 21 common frames omitted
15:48:28,049 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@44:94 - ActionException in Action for tag [appender] ch.qos.logback.core.joran.spi.ActionException: ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender
        at ch.qos.logback.core.joran.spi.ActionException: ch.qos.logback.core.util.DynamicClassLoadingException: Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender

Here where i've put the jar 

[root@devnode016 lib]# ls /var/opt/hosting/nifi-1.1.0/lib/logstash-logback-encoder-4.7.jar
/var/opt/hosting/nifi-1.1.0/lib/logstash-logback-encoder-4.7.jar
3,255 Views
0 Kudos

avatar
author-rank bbende author-rank
Master Guru

Created ‎02-02-2017 03:26 PM

Not really sure about that one, can you provide the part of logback.xml where you declared the new appender?

3,255 Views
0 Kudos

avatar
author-rank maykiwogno
Rising Star

Created ‎02-03-2017 02:18 PM

Hi all,

I'll found that some wrong. So if you need use logstash-encoder with NIFI.

Make these modifications :

Links jackson jar from bootstrap to lib 

lrwxrwxrwx 1 root root       71 Feb  3 15:04 jackson-annotations-2.6.0.jar -> lib/bootstrap/jackson-annotations-2.6.0.jar
lrwxrwxrwx 1 root root       64 Feb  3 15:04 jackson-core-2.6.1.jar -> lib/bootstrap/jackson-core-2.6.1.jar
lrwxrwxrwx 1 root root       68 Feb  3 15:04 jackson-databind-2.6.1.jar -> lib/bootstrap/jackson-databind-2.6.1.jar

Add logstash-logback-encoder-4.7.jar to lib

and configure logback.xml with Appender.

3,255 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streams Messaging - Kubernetes Operator ...
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
View More Announcements