Atlas generating huge log files

[#1]

HDP 2.3.2 cluster is generating huge log files Atlas, see this example:

[root@APHDPMND01 atlas]# ls -la
drwxr-xr-x.  2 atlas hadoop       4096 Nov 19 00:00 .
drwxr-xr-x. 35 root  root         4096 Nov 15 03:07 ..
-rw-r--r--.  1 atlas hadoop 1548523843 Oct 29 16:52 metadata.20151020-105724.err
-rw-r--r--.  1 atlas hadoop          0 Oct 20 10:57 metadata.20151020-105724.out
-rw-r--r--.  1 atlas hadoop 1520931501 Nov 11 11:58 metadata.20151102-100301.err
-rw-r--r--.  1 atlas hadoop          0 Nov  2 10:03 metadata.20151102-100301.out
-rw-r--r--.  1 atlas hadoop  978775361 Nov 19 17:48 metadata.20151113-153007.err
-rw-r--r--.  1 atlas hadoop          0 Nov 13 15:30 metadata.20151113-153007.out

I see about 15,000 lines like below per second in the log files. Is it a known issue? Any solution?

Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class java.lang.String
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve
SEVERE: null
java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
	at java.lang.Class.newInstance(Class.java:368)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
	at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
	at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
	at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:179)
	at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:125)
	at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:67)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[#2]

@Guilherme Braccialli I reached out to Andrew regarding this issue yesterday, havent heard from him. I'll ping him again and open a support ticket. I need Atlas for a Demo so a patch would be nice 🙂

Keep you posted.

[#3]

@Jonas Straub @Guilherme Braccialli

Never know ..all related to each other 🙂

[#4 · Accepted solution]

This seems to be an issue with jersey version 1.9. http://comments.gmane.org/gmane.comp.java.jersey.u...

This is fixed in Atlas 0.6 version(https://github.com/apache/incubator-atlas/blob/bra...) but not on Atlas 0.5 version(https://github.com/apache/incubator-atlas/blob/bra...

Issue is still seen in hadoop 2.7.0 because jersey version is still pointing to version 1.9 (https://github.com/apache/hadoop/blob/branch-2.7.0...

I am reaching out to the atlas team on having this fix included in the next 2.3.x maintenance release.

[#5]

Thanks, that would be great!

[#6]

Thank you @Ayub Pathan. Is is possible to provide a fix?

[#7]

Yes @Guilherme Braccialli, I reached out to the atlas team to have this fix included in the next maintenance release. I will followup with the team and keep posted.

[#8]

I mean a hot fix to be applied now to hdp 2.3.2 installation before upgrade.

[#9]

I am not sure hot-fix will be possible here because change in the jersey version triggers change in all its dependencies.

One good news is, this fix is merged to the next maintenance release today.

[#10]

@Ayub Pathan Has the fix already been released?

[#11]

@Eyad Garelnabi Yes, fix is available in HDP-2.3.4.2 maintenance release.