Can't update services configuration in Ambari when accessing via Knox.

jakub_igla1 — Tue, 12 Jun 2018 21:23:00 GMT

Hi,

I've installed Apache Knox via Ambari.
Added Ambari service + configured Ldap.

Based on that, I was able to curl webhdfs as specific ldap use.
But when I access Ambari via knox in a Browser I can browse everything, but can't make changes to configuration.
Every time I change some configs in services I get this ajax response:

{
  "status" : 400,
  "message" : "Invalid Request: Malformed Request Body.  An exception occurred parsing the request body: Unexpected character ('%' (code 37)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: java.io.StringReader@3af8ede7; line: 1, column: 3]"
}

When I access it directly, all works as expected.

Worth to mention, that I have nginx on top of knox (I need it for many reasons).
And my basic configuration is as follows:

server {
   listen 443 ssl;
   server_name gateway.example.com;
   ssl_certificate /etc/ssl/cert.crt;
   ssl_certificate_key /etc/ssl/cert.key;
   location / {
     proxy_pass https://knox:8443;

     # this magic is needed for WebSocket
     proxy_http_version  1.1;
     proxy_redirect      off;
     proxy_set_header    Upgrade $http_upgrade;
     proxy_set_header    Connection "upgrade";
     proxy_set_header    Host $http_host;
   }
}

In my browser, I access Ambari like that:

https://gateway.example.com/gateway/default/ambari

Another thing that doesn't work for me regarding this matter is when I set this value:

default.app.topology.name to 'default' (as above url indicates).

I always get 500 error and some weird logs about rewriting.

Happy to provide more info, but not sure what else.

Re: Can't update services configuration in Ambari when accessing via Knox.

jakub_igla1 — Tue, 12 Jun 2018 21:35:07 GMT

Okay, I've resolved that first issues (with config changes failures) by adding this to my topology:

<service>        <role>AMBARI</role>        <url>http://hostname:port</url>    </service>

before I only had AMBARIUI service. Which now all makes sense.

But still, can't setup a default gateway so my urls are shorter.

Re: Can't update services configuration in Ambari when accessing via Knox.

jakub_igla1 — Tue, 12 Jun 2018 21:54:02 GMT

Very sorry to spam you with so many messages, but I'm attaching knox logs

==> /var/log/knox/gateway-audit.log <==
18/06/12 14:51:06 ||562e17ec-da4e-4c6f-8207-2e1ecfc149bb|audit|10.251.3.4|AMBARIUI||||access|uri|/gateway/default/ambari/ambari|unavailable|Request method: GET
18/06/12 14:51:06 ||562e17ec-da4e-4c6f-8207-2e1ecfc149bb|audit|10.251.3.4|AMBARIUI|anonymous|||authentication|uri|/gateway/default/ambari/ambari|success|


==> /var/log/knox/gateway.log <==
2018-06-12 14:51:06,681 INFO  hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true


==> /var/log/knox/gateway-audit.log <==
18/06/12 14:51:06 ||562e17ec-da4e-4c6f-8207-2e1ecfc149bb|audit|10.251.3.4|AMBARIUI|anonymous|||authorization|uri|/gateway/default/ambari/ambari|success|


==> /var/log/knox/gateway.log <==
2018-06-12 14:51:06,682 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL: https://gateway.example.com:443/ambari, direction: IN via rule: AMBARIUI/ambari/inbound/root, status: FAILUREexample.com
2018-06-12 14:51:06,682 ERROR hadoop.gateway (UrlRewriteProcessor.java:rewrite(169)) - Failed to rewrite URL: https://gateway.example.com:443/ambari, direction: IN via rule: AMBARIUI/ambari/inbound/root, status: FAILURE


==> /var/log/knox/gateway-audit.log <==
18/06/12 14:51:06 ||562e17ec-da4e-4c6f-8207-2e1ecfc149bb|audit|10.251.3.4|AMBARIUI|anonymous|||dispatch|uri||unavailable|Request method: GET
18/06/12 14:51:06 ||562e17ec-da4e-4c6f-8207-2e1ecfc149bb|audit|10.251.3.4|AMBARIUI|anonymous|||dispatch|uri||failure|


==> /var/log/knox/gateway.log <==
2018-06-12 14:51:06,684 WARN  hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(147)) - Connection exception dispatching request:  org.apache.http.client.ClientProtocolException
org.apache.http.client.ClientProtocolException
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:130)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:116)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:278)
	at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:124)
	at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:107)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.AclsAuthorizationFilter.doFilter(AclsAuthorizationFilter.java:89)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter$1.run(AnonymousAuthFilter.java:76)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter.continueWithEstablishedSecurityContext(AnonymousAuthFilter.java:71)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter.doFilter(AnonymousAuthFilter.java:66)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:30)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:139)
	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:91)
	at org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:141)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.apache.hadoop.gateway.filter.DefaultTopologyHandler.handle(DefaultTopologyHandler.java:84)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.ProtocolException: Target host is not specified
	at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71)
	at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	... 54 more
2018-06-12 14:51:06,685 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(63)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2018-06-12 14:51:06,685 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(63)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2018-06-12 14:51:06,685 ERROR hadoop.gateway (AbstractGatewayFilter.java:doFilter(63)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2018-06-12 14:51:06,686 ERROR hadoop.gateway (GatewayFilter.java:doFilter(141)) - Gateway processing failed: java.io.IOException: Service connectivity error.
java.io.IOException: Service connectivity error.
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:148)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:116)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:278)
	at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:124)
	at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:107)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.AclsAuthorizationFilter.doFilter(AclsAuthorizationFilter.java:89)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter$1.run(AnonymousAuthFilter.java:76)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter.continueWithEstablishedSecurityContext(AnonymousAuthFilter.java:71)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter.doFilter(AnonymousAuthFilter.java:66)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:30)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:139)
	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:91)
	at org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:141)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.apache.hadoop.gateway.filter.DefaultTopologyHandler.handle(DefaultTopologyHandler.java:84)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:748)


==> /var/log/knox/gateway-audit.log <==
18/06/12 14:51:06 ||562e17ec-da4e-4c6f-8207-2e1ecfc149bb|audit|10.251.3.4|AMBARIUI|anonymous|||access|uri|/gateway/default/ambari/ambari|failure|


==> /var/log/knox/gateway.log <==
2018-06-12 14:51:06,686 ERROR hadoop.gateway (GatewayServlet.java:service(143)) - Gateway processing failed: java.io.IOException: Service connectivity error.
java.io.IOException: Service connectivity error.
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:148)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:116)
	at org.apache.hadoop.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:278)
	at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:124)
	at org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:107)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.AclsAuthorizationFilter.doFilter(AclsAuthorizationFilter.java:89)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:60)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter$1.run(AnonymousAuthFilter.java:76)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter.continueWithEstablishedSecurityContext(AnonymousAuthFilter.java:71)
	at org.apache.hadoop.gateway.filter.AnonymousAuthFilter.doFilter(AnonymousAuthFilter.java:66)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:30)
	at org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
	at org.apache.hadoop.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:332)
	at org.apache.hadoop.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:232)
	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:139)
	at org.apache.hadoop.gateway.GatewayFilter.doFilter(GatewayFilter.java:91)
	at org.apache.hadoop.gateway.GatewayServlet.service(GatewayServlet.java:141)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.apache.hadoop.gateway.filter.DefaultTopologyHandler.handle(DefaultTopologyHandler.java:84)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:748)

Re: Can't update services configuration in Ambari when accessing via Knox.

jsensharma — Wed, 13 Jun 2018 05:29:51 GMT

@Jakub Igla

While setting up knox for ambari we must define both the following roled inside the knos topology:

   <service>
        <role>AMBARI</role>
        <url>http://$AMBARI_HOST:8080</url>
    </service>
    <service>
        <role>AMBARIUI</role>
        <url>http://$AMBARI_HOST:8080</url>
    </service>

After making the above changes please try to restart the knox and then try again.

Re: Can't update services configuration in Ambari when accessing via Knox.

jakub_igla1 — Wed, 13 Jun 2018 13:10:50 GMT

Yes, the Ambari role was missing.
Do you know how to get rid of /gateway/default from the url based on the logs I provided above? Many thanks!

question Re: Can't update services configuration in Ambari when accessing via Knox. in Archives of Support Questions (Read Only)

Can't update services configuration in Ambari when accessing via Knox.

Re: Can't update services configuration in Ambari when accessing via Knox.

Re: Can't update services configuration in Ambari when accessing via Knox.

Re: Can't update services configuration in Ambari when accessing via Knox.

Re: Can't update services configuration in Ambari when accessing via Knox.