Atlas generating huge log files

[#1]

HDP 2.3.2 cluster is generating huge log files Atlas, see this example:

[root@APHDPMND01 atlas]# ls -la
drwxr-xr-x.  2 atlas hadoop       4096 Nov 19 00:00 .
drwxr-xr-x. 35 root  root         4096 Nov 15 03:07 ..
-rw-r--r--.  1 atlas hadoop 1548523843 Oct 29 16:52 metadata.20151020-105724.err
-rw-r--r--.  1 atlas hadoop          0 Oct 20 10:57 metadata.20151020-105724.out
-rw-r--r--.  1 atlas hadoop 1520931501 Nov 11 11:58 metadata.20151102-100301.err
-rw-r--r--.  1 atlas hadoop          0 Nov  2 10:03 metadata.20151102-100301.out
-rw-r--r--.  1 atlas hadoop  978775361 Nov 19 17:48 metadata.20151113-153007.err
-rw-r--r--.  1 atlas hadoop          0 Nov 13 15:30 metadata.20151113-153007.out

I see about 15,000 lines like below per second in the log files. Is it a known issue? Any solution?

Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator attachTypes
INFO: Couldn't find JAX-B element for class java.lang.String
Nov 12, 2015 3:40:04 PM com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 resolve
SEVERE: null
java.lang.IllegalAccessException: Class com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can not access a member of class javax.ws.rs.core.Response with modifiers "protected"
	at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
	at java.lang.Class.newInstance(Class.java:368)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
	at com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
	at com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
	at com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
	at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:179)
	at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:125)
	at com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:67)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
	at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
	at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[#2]

Without spending too much time, it appears to be a defect to me.

@Balu any thoughts?

[#3]

Unfortunately, I don't have any thoughts. We should follow up with Atlas. Qq : Does this happen with Falcon/Oozie logs as well?

[#4]

no. actually customer is not using falcon, oozie not even atlas.

[#5]

I havent seen any Atlas error in my Falcon log so far

[#6]

@Guilherme Braccialli Please see this https://issues.apache.org/jira/browse/HADOOP-11461

[#7]

Interesting, according to ATLAS-16 this issue has been fixed, but according to HADOOP-11461 its still open ....

[#8]

@Neeraj Sabharwal Its not the same error. The exception stack trace pasted by OP is originating with Atlas (org.apache.atlas.web.filters.AuditFilter.doFilter) where as the one in the JIRA is within Hadoop. Same exception class different applications.

[#9]

I have seen this in my Atlas logs as well. See this https://issues.apache.org/jira/browse/ATLAS-16

Apparently this issue was already fixed back in August, I dont know why it was not released with 2.3.2.

[#10]

@Jonas Straub do you know if we have a hot fix for this? Customer log directory is getting full quickly.

[#11]

@Guilherme Braccialli I reached out to Andrew regarding this issue yesterday, havent heard from him. I'll ping him again and open a support ticket. I need Atlas for a Demo so a patch would be nice 🙂

Keep you posted.

[#12]

@Jonas Straub @Guilherme Braccialli

Never know ..all related to each other 🙂

[#13 · Accepted solution]

This seems to be an issue with jersey version 1.9. http://comments.gmane.org/gmane.comp.java.jersey.u...

This is fixed in Atlas 0.6 version(https://github.com/apache/incubator-atlas/blob/bra...) but not on Atlas 0.5 version(https://github.com/apache/incubator-atlas/blob/bra...

Issue is still seen in hadoop 2.7.0 because jersey version is still pointing to version 1.9 (https://github.com/apache/hadoop/blob/branch-2.7.0...

I am reaching out to the atlas team on having this fix included in the next 2.3.x maintenance release.

[#14]

Thanks, that would be great!

[#15]

Thank you @Ayub Pathan. Is is possible to provide a fix?

[#16]

Yes @Guilherme Braccialli, I reached out to the atlas team to have this fix included in the next maintenance release. I will followup with the team and keep posted.

[#17]

I mean a hot fix to be applied now to hdp 2.3.2 installation before upgrade.

[#18]

I am not sure hot-fix will be possible here because change in the jersey version triggers change in all its dependencies.

One good news is, this fix is merged to the next maintenance release today.

[#19]

@Ayub Pathan Has the fix already been released?

[#20]

@Eyad Garelnabi Yes, fix is available in HDP-2.3.4.2 maintenance release.

[#21]

@Eyad Garelnabi I'm still seeing this exact issue in a fresh install of 2.3.4.7. Can you confirm this was actually released in 2.3.2?