knox with yarnui - colon (:) character becomes %3A

[#1]

Hi Experts,

Please, any help in getting past this issue is appreciated.

We are using HDP 2.6.1. Though KNOX with YARNUI is not officially supported, I thought I would give it a try by modifying the service and rewrite xml files a bit.

It seems to be working fine except a few places. One of the issues that I am facing is that it is placing %3A instead of a colon (:) in the request URL. Here's the process flow.

1) Initial page - opens fine

https://knox.sandbox.com:8443/gateway/default/yarn/

2) Click on a finished Tez application - opens fine

https://knox.sandbox.com:8443/gateway/default/yarn/cluster/app/application_1544594647974_0003

3) Click on logs - breaks. Here's what the link looks like.

https://knox.sandbox.com:8443/gateway/default/yarn/nodemanager/node/containerlogs/container_e212_154...

After clicking the link, the URL in the address bar becomes as below (Colon becomes %3A)

http://rm.sandbox.com:19888/jobhistory/logs/datanode03.sandbox.com%3A45454/container_e212_1544594647...

And the error shown on the web page is

Cannot get container logs. Invalid nodeId: datanode03.sandbox.com%3A45454

Gateway log shows these lines

2018-12-14 03:51:26,968 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: http://rm.sandbox.com:8088/proxy/application_1544594647974_0003/, direction: OUT via explicit rule: YARNUI/yarn/outbound/apps/history to URL: https://knox.sandbox.com:8443/gateway/default/yarn/proxy/application_1544594647974_0003
2018-12-14 03:51:26,969 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: logs, direction: OUT via implicit rule: YARNUI/yarn/outbound/logs to URL: https://knox.sandbox.com:8443/gateway/default/yarn/logs
2018-12-14 03:51:26,970 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: http://datanode03.sandbox.com:8042/node/containerlogs/container_e212_1544594647974_0003_01_000001/hi..., direction: OUT via explicit rule: YARNUI/yarn/outbound/node/containerlogs to URL: https://knox.sandbox.com:8443/gateway/default/yarn/nodemanager/node/containerlogs/container_e212_154...
2018-12-14 03:51:26,970 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: /cluster/app, direction: OUT via explicit rule: YARNUI/yarn/outbound/apps/app to URL: https://knox.sandbox.com:8443/gateway/default/yarn/cluster/app
2018-12-14 03:52:47,645 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /yarn/nodemanager/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive
2018-12-14 03:52:47,650 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://knox.sandbox.com:8443/gateway/default/yarn/nodemanager/node/containerlogs/container_e212_154..., direction: IN via implicit rule: YARNUI/yarn/inbound/node/containerlogs to URL: http://datanode03.sandbox.com:8042/node/containerlogs/container_e212_1544594647974_0003_01_000001/hi...
2018-12-14 03:52:47,650 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://knox.sandbox.com:8443/gateway/default/yarn/cluster/app/application_1544594647974_0003, direction: IN via implicit rule: YARNUI/yarn/inbound/cluster to URL: http://rm.sandbox.com:8088/cluster/app/application_1544594647974_0003
2018-12-14 03:52:47,650 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request: GET http://datanode03.sandbox.com:8042/node/containerlogs/container_e212_1544594647974_0003_01_000001/hi...
2018-12-14 03:52:47,666 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response status: 307
2018-12-14 03:52:47,667 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit character set UTF-8 for entity of type text/plain
2018-12-14 03:52:47,667 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound response entity content type: text/plain; charset=utf-8

[#2 · Accepted solution]

This is most likely fixed in Apache Knox 1.2.0 (KNOX-1207) and should be fixed in HDP 3.1 just released.

[#3]

Thank You Kevin. Unfortunately, an upgrade does not seem to be in the cards as of now. Would downloading the latest service directories containing the latest service and rewrite xml be able to fix it instead?

[#4]

Potentially but no guarantees. You would lose those changes on upgrades and would be up to you to keep them in sync. I know it is something that will not be supported but you seem to acknowledge that in the question with "Though KNOX with YARNUI is not officially supported"

[#5]

@Kevin Risden Thank You.