question Atlas generating huge log files in Support Questions

Atlas generating huge log files

gbraccialli3 — Fri, 04 Dec 2015 05:30:39 GMT

HDP 2.3.2 cluster is generating huge log files Atlas, see this example:

[root@APHDPMND01 atlas]# ls -la
drwxr-xr-x.  2 atlas hadoop       4096 Nov 19 00:00 .
drwxr-xr-x. 35 root  root         4096 Nov 15 03:07 ..
-rw-r--r--.  1 atlas hadoop 1548523843 Oct 29 16:52 metadata.20151020-105724.err
-rw-r--r--.  1 atlas hadoop          0 Oct 20 10:57 metadata.20151020-105724.out
-rw-r--r--.  1 atlas hadoop 1520931501 Nov 11 11:58 metadata.20151102-100301.err
-rw-r--r--.  1 atlas hadoop          0 Nov  2 10:03 metadata.20151102-100301.out
-rw-r--r--.  1 atlas hadoop  978775361 Nov 19 17:48 metadata.20151113-153007.err
-rw-r--r--.  1 atlas hadoop          0 Nov 13 15:30 metadata.20151113-153007.out

I see about 15,000 lines like below per second in the log files. Is it a known issue? Any solution?

Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class java.lang.String
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve
SEVERE: null
java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
	at java.lang.Class.newInstance(Class.java:368)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
	at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
	at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
	at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:179)
	at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:125)
	at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:67)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Re: Atlas generating huge log files

bsaini — Fri, 04 Dec 2015 05:45:03 GMT

Without spending too much time, it appears to be a defect to me.

@Balu any thoughts?

Re: Atlas generating huge log files

nsabharwal — Fri, 04 Dec 2015 05:55:13 GMT

@Guilherme Braccialli Please see this https://issues.apache.org/jira/browse/HADOOP-11461

Re: Atlas generating huge log files

jstraub — Fri, 04 Dec 2015 05:56:27 GMT

I have seen this in my Atlas logs as well. See this https://issues.apache.org/jira/browse/ATLAS-16

Apparently this issue was already fixed back in August, I dont know why it was not released with 2.3.2.

Re: Atlas generating huge log files

jstraub — Fri, 04 Dec 2015 05:57:56 GMT

Interesting, according to ATLAS-16 this issue has been fixed, but according to HADOOP-11461 its still open ....

Re: Atlas generating huge log files

gbraccialli3 — Fri, 04 Dec 2015 06:02:40 GMT

@Jonas Straub do you know if we have a hot fix for this? Customer log directory is getting full quickly.

Re: Atlas generating huge log files

jstraub — Fri, 04 Dec 2015 06:07:27 GMT

@Guilherme Braccialli I reached out to Andrew regarding this issue yesterday, havent heard from him. I'll ping him again and open a support ticket. I need Atlas for a Demo so a patch would be nice 🙂

Keep you posted.

Re: Atlas generating huge log files

bvellanki — Fri, 04 Dec 2015 06:37:57 GMT

Unfortunately, I don't have any thoughts. We should follow up with Atlas. Qq : Does this happen with Falcon/Oozie logs as well?

Re: Atlas generating huge log files

gbraccialli3 — Fri, 04 Dec 2015 06:40:09 GMT

no. actually customer is not using falcon, oozie not even atlas.

Re: Atlas generating huge log files

jstraub — Fri, 04 Dec 2015 06:47:20 GMT

I havent seen any Atlas error in my Falcon log so far

Re: Atlas generating huge log files

nsabharwal — Fri, 04 Dec 2015 06:52:09 GMT

@Jonas Straub @Guilherme Braccialli

Never know ..all related to each other 🙂

Re: Atlas generating huge log files

bsaini — Sat, 05 Dec 2015 04:16:32 GMT

@Neeraj Sabharwal Its not the same error. The exception stack trace pasted by OP is originating with Atlas (org.apache.atlas.web.filters.AuditFilter.doFilter) where as the one in the JIRA is within Hadoop. Same exception class different applications.

Re: Atlas generating huge log files

apathan — Thu, 10 Dec 2015 15:30:02 GMT

This seems to be an issue with jersey version 1.9. http://comments.gmane.org/gmane.comp.java.jersey.u...

This is fixed in Atlas 0.6 version(https://github.com/apache/incubator-atlas/blob/bra...) but not on Atlas 0.5 version(https://github.com/apache/incubator-atlas/blob/bra...

Issue is still seen in hadoop 2.7.0 because jersey version is still pointing to version 1.9 (https://github.com/apache/hadoop/blob/branch-2.7.0...

I am reaching out to the atlas team on having this fix included in the next 2.3.x maintenance release.

Re: Atlas generating huge log files

jstraub — Thu, 10 Dec 2015 15:32:27 GMT

Thanks, that would be great!

Re: Atlas generating huge log files

gbraccialli3 — Thu, 10 Dec 2015 18:46:43 GMT

Thank you @Ayub Pathan. Is is possible to provide a fix?

Re: Atlas generating huge log files

apathan — Thu, 10 Dec 2015 19:59:02 GMT

Yes @Guilherme Braccialli, I reached out to the atlas team to have this fix included in the next maintenance release. I will followup with the team and keep posted.

Re: Atlas generating huge log files

gbraccialli3 — Thu, 10 Dec 2015 20:03:13 GMT

I mean a hot fix to be applied now to hdp 2.3.2 installation before upgrade.

Re: Atlas generating huge log files

apathan — Fri, 11 Dec 2015 18:32:17 GMT

I am not sure hot-fix will be possible here because change in the jersey version triggers change in all its dependencies.

One good news is, this fix is merged to the next maintenance release today.

Re: Atlas generating huge log files

egarelnabi — Wed, 23 Mar 2016 18:18:25 GMT

@Ayub Pathan Has the fix already been released?

Re: Atlas generating huge log files

egarelnabi — Wed, 23 Mar 2016 18:20:32 GMT

@Guilherme Braccialli Were you able to locate a fix for 2.3.x? I'm facing the same issue