Issue accessing WEBHDFS using Knox authentication - ERROR 500

[#1]

Executing below curl command

curl -iku guest:guest-password -X GET 'https://hd-poc-06.unix.gsm1900.org:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS'

Throwing this error

HTTP/1.1 500 Server Error Set-Cookie: JSESSIONID=1jlx4cirafr7e1qdcrk35ixxrz;Path=/gateway/default;Secure;HttpOnly Content-Type: text/html;charset=ISO-8859-1 Cache-Control: must-revalidate,no-cache,no-store Content-Length: 1398 Server: Jetty(8.1.14.v20131031) <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body><h2>HTTP ERROR 500</h2> <p>Problem accessing /gateway/default/webhdfs/v1/. Reason: <pre> Server Error</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> <br/> <br/>

[#2]

Could you provide the output of the following command while executing the curl command?

tail -f /var/log/knox/gateway*.log

Also let us know the exact HDP version you're using, and whether you are using Kerberos and/or NameNode HA.

[#3]

@Alex Miller I see below error messages,

Caused by: java.io.IOException: java.io.IOException: Service connectivity error. at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.failoverRequest(WebHdfsHaDispatch.java:153) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.executeRequest(WebHdfsHaDispatch.java:102) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.failoverRequest(WebHdfsHaDispatch.java:147) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.executeRequest(WebHdfsHaDispatch.java:102) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.failoverRequest(WebHdfsHaDispatch.java:147) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.executeRequest(WebHdfsHaDispatch.java:102) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.failoverRequest(WebHdfsHaDispatch.java:147) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.executeRequest(WebHdfsHaDispatch.java:102) at org.apache.hadoop.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:294) at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:145) at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:117) at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.filter.AclsAuthorizationFilter.doFilter(AclsAuthorizationFilter.java:89) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:195) at org.apache.hadoop.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:152) at org.apache.hadoop.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:66) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60) at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93) at org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90) ... 59 more Caused by: java.io.IOException: Service connectivity error. at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:140) at org.apache.hadoop.gateway.hdfs.dispatch.WebHdfsHaDispatch.executeRequest(WebHdfsHaDispatch.java:92) ... 85 more

[#4]

@Alex Miller Yes it is NN HA enabled cluster with Kerberos authentication.

[#5]

2016-05-24 09:28:45,770 ERROR hadoop.gateway (GatewayServlet.java:service(126)) - Gateway processing failed: javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: java.io.IOException: Service connectivity error. javax.servlet.ServletException: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: java.io.IOException: Service connectivity error. at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196) at org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.filter.ResponseCookieFilter.doFilter(ResponseCookieFilter.java:38) at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:30) at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61) at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:315) at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:215) at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:129) at org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:121) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.shiro.subject.ExecutionException: java.security.PrivilegedActionException: java.io.IOException: java.io.IOException: Service connectivity error. at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:385) at org.apache.hadoop.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:74)

[#6]

is ranger enabled for Knox? If so make sure you create a ranger knox policy and provide permission for guest user.

[#7 · Accepted solution]

@Alex Miller @Pranay Vyas Thanks for your support.

Issue is resolved by correcting NameNode details,

<service> <role>NAMENODE</role> <url>nameservice1</url> </service>

Earlier I mentioned url as hdfs://nameservice1 updated to nameservice1.