Atlas generating huge log files

[#1]

HDP 2.3.2 cluster is generating huge log files Atlas, see this example:

[root@APHDPMND01 atlas]# ls -la
drwxr-xr-x.  2 atlas hadoop       4096 Nov 19 00:00 .
drwxr-xr-x. 35 root  root         4096 Nov 15 03:07 ..
-rw-r--r--.  1 atlas hadoop 1548523843 Oct 29 16:52 metadata.20151020-105724.err
-rw-r--r--.  1 atlas hadoop          0 Oct 20 10:57 metadata.20151020-105724.out
-rw-r--r--.  1 atlas hadoop 1520931501 Nov 11 11:58 metadata.20151102-100301.err
-rw-r--r--.  1 atlas hadoop          0 Nov  2 10:03 metadata.20151102-100301.out
-rw-r--r--.  1 atlas hadoop  978775361 Nov 19 17:48 metadata.20151113-153007.err
-rw-r--r--.  1 atlas hadoop          0 Nov 13 15:30 metadata.20151113-153007.out

I see about 15,000 lines like below per second in the log files. Is it a known issue? Any solution?

Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class java.lang.String
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve
SEVERE: null
java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
	at java.lang.Class.newInstance(Class.java:368)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
	at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
	at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
	at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:179)
	at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:125)
	at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:67)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[#2]

Without spending too much time, it appears to be a defect to me.

@Balu any thoughts?

[#3]

Unfortunately, I don't have any thoughts. We should follow up with Atlas. Qq : Does this happen with Falcon/Oozie logs as well?

[#4]

no. actually customer is not using falcon, oozie not even atlas.

[#5]

I havent seen any Atlas error in my Falcon log so far

[#6]

@Guilherme Braccialli Please see this https://issues.apache.org/jira/browse/HADOOP-11461

[#7]

Interesting, according to ATLAS-16 this issue has been fixed, but according to HADOOP-11461 its still open ....

[#8]

@Neeraj Sabharwal Its not the same error. The exception stack trace pasted by OP is originating with Atlas (org.apache.atlas.web.filters.AuditFilter.doFilter) where as the one in the JIRA is within Hadoop. Same exception class different applications.

[#9]

I have seen this in my Atlas logs as well. See this https://issues.apache.org/jira/browse/ATLAS-16

Apparently this issue was already fixed back in August, I dont know why it was not released with 2.3.2.

[#10]

@Jonas Straub do you know if we have a hot fix for this? Customer log directory is getting full quickly.