Member since
04-20-2017
5
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3979 | 05-05-2017 12:48 PM |
05-05-2017
12:48 PM
1 Kudo
I finally got it to work!
The problem had to do with that sss was configured to use fully qualified names.
I could either change the hadoop.security.auth_to_local in core-site.xml -> to not convert myaduser@NET.LOCAL to myaduser. This would make my hdfs user to myaduser@NET.LOCAL and I could see the logs.
The alternative that we went for is to change the following settings in /etc/sssd/sssd.conf Remove: default_domain_suffix = net.local
Change: use_fully_qualified_names = False And run on each machine:
systemctl restart sssd
sss_cache -E
... View more
05-04-2017
09:35 AM
$ hadoop org.apache.hadoop.security.HadoopKerberosName myaduser
Name: myaduser to myaduser
$hadoop org.apache.hadoop.security.HadoopKerberosName myaduser@NET.LOCAL
Name: myaduser@NET.LOCAL to myaduser
... View more
04-28-2017
09:27 AM
Hi, I am trying to read logs from jobs started by an AD user, but I get the following error message (both in the web UI and using yarn logs -applicationId <id>): Exception reading log file. Application submitted by 'myaduser' doesn't own requested log file : directory.info I can however read data from system users (hive, zeppelin etc.). When I look at the nodemanager logs I see the following error: java.io.IOException: Owner 'myaduser@net.local' for path /var/log/hadoop/yarn/log/application_1493304120821_0017/container_e38_1493304120821_0017_01_000001/directory.info did not match expected owner 'myaduser'
at org.apache.hadoop.io.SecureIOUtils.checkStat(SecureIOUtils.java:285)
at org.apache.hadoop.io.SecureIOUtils.forceSecureOpenForRead(SecureIOUtils.java:219)
at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:204)
at org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerLogsUtils.openLogFileForRead(ContainerLogsUtils.java:170)
at org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerLogsPage$ContainersLogsBlock.printLogFile(ContainerLogsPage.java:135)
at org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerLogsPage$ContainersLogsBlock.render(ContainerLogsPage.java:109)
at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
at org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
at org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
at org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.logs(NMController.java:70)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebAppFilter.doFilter(NMWebAppFilter.java:72)
at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1409)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
This occurs when I run both hive queries and Spark through spark-submit and Zeppelin with livy (impersonation). /var/log/hadoop/yarn/log/application_1493304120821_0017/container_e38_1493304120821_0017_01_000001/directory.info is owned by myaduser@net.local:hadoop. myaduser@net.local is not in the group hadoop however. In all services we are using only the username, not with the realm. For example the hadoop user is only "myaduser" (not myaduser@NET.LOCAL and the home folder is /user/myaduser/.
... View more
Labels:
- Labels:
-
Apache YARN