Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Atlas generating huge log files

Solved Go to solution

Atlas generating huge log files

HDP 2.3.2 cluster is generating huge log files Atlas, see this example:

[root@APHDPMND01 atlas]# ls -la
drwxr-xr-x.  2 atlas hadoop       4096 Nov 19 00:00 .
drwxr-xr-x. 35 root  root         4096 Nov 15 03:07 ..
-rw-r--r--.  1 atlas hadoop 1548523843 Oct 29 16:52 metadata.20151020-105724.err
-rw-r--r--.  1 atlas hadoop          0 Oct 20 10:57 metadata.20151020-105724.out
-rw-r--r--.  1 atlas hadoop 1520931501 Nov 11 11:58 metadata.20151102-100301.err
-rw-r--r--.  1 atlas hadoop          0 Nov  2 10:03 metadata.20151102-100301.out
-rw-r--r--.  1 atlas hadoop  978775361 Nov 19 17:48 metadata.20151113-153007.err
-rw-r--r--.  1 atlas hadoop          0 Nov 13 15:30 metadata.20151113-153007.out

I see about 15,000 lines like below per second in the log files. Is it a known issue? Any solution?

Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class java.lang.String
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve
SEVERE: null
java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
	at java.lang.Class.newInstance(Class.java:368)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
	at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
	at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
	at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:179)
	at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:125)
	at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:67)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


1 ACCEPTED SOLUTION

Accepted Solutions

Re: Atlas generating huge log files

This seems to be an issue with jersey version 1.9. http://comments.gmane.org/gmane.comp.java.jersey.u...

This is fixed in Atlas 0.6 version(https://github.com/apache/incubator-atlas/blob/bra...) but not on Atlas 0.5 version(https://github.com/apache/incubator-atlas/blob/bra...

Issue is still seen in hadoop 2.7.0 because jersey version is still pointing to version 1.9 (https://github.com/apache/hadoop/blob/branch-2.7.0...

I am reaching out to the atlas team on having this fix included in the next 2.3.x maintenance release.

23 REPLIES 23

Re: Atlas generating huge log files

Without spending too much time, it appears to be a defect to me.

@Balu any thoughts?

Re: Atlas generating huge log files

Rising Star

Unfortunately, I don't have any thoughts. We should follow up with Atlas. Qq : Does this happen with Falcon/Oozie logs as well?

Re: Atlas generating huge log files

no. actually customer is not using falcon, oozie not even atlas.

Re: Atlas generating huge log files

I havent seen any Atlas error in my Falcon log so far

Re: Atlas generating huge log files

Re: Atlas generating huge log files

Interesting, according to ATLAS-16 this issue has been fixed, but according to HADOOP-11461 its still open ....

Re: Atlas generating huge log files

@Neeraj Sabharwal Its not the same error. The exception stack trace pasted by OP is originating with Atlas (org.apache.atlas.web.filters.AuditFilter.doFilter) where as the one in the JIRA is within Hadoop. Same exception class different applications.

Re: Atlas generating huge log files

I have seen this in my Atlas logs as well. See this https://issues.apache.org/jira/browse/ATLAS-16

Apparently this issue was already fixed back in August, I dont know why it was not released with 2.3.2.

Re: Atlas generating huge log files

@Jonas Straub do you know if we have a hot fix for this? Customer log directory is getting full quickly.