knox with yarnui - colon (:) character becomes %3A

manishsingh2k — Fri, 14 Dec 2018 21:05:13 GMT

Hi Experts,

Please, any help in getting past this issue is appreciated.

We are using HDP 2.6.1. Though KNOX with YARNUI is not officially supported, I thought I would give it a try by modifying the service and rewrite xml files a bit.

It seems to be working fine except a few places. One of the issues that I am facing is that it is placing %3A instead of a colon (:) in the request URL. Here's the process flow.

1) Initial page - opens fine

https://knox.sandbox.com:8443/gateway/default/yarn/

2) Click on a finished Tez application - opens fine

https://knox.sandbox.com:8443/gateway/default/yarn/cluster/app/application_1544594647974_0003

3) Click on logs - breaks. Here's what the link looks like.

https://knox.sandbox.com:8443/gateway/default/yarn/nodemanager/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive?host=datanode03.sandbox.com&port=8042

After clicking the link, the URL in the address bar becomes as below (Colon becomes %3A)

http://rm.sandbox.com:19888/jobhistory/logs/datanode03.sandbox.com%3A45454/container_e212_1544594647974_0003_01_000001/container_e212_1544594647974_0003_01_000001/hive?doAs=myuser

And the error shown on the web page is

Cannot get container logs. Invalid nodeId: datanode03.sandbox.com%3A45454

Gateway log shows these lines

2018-12-14 03:51:26,968 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: http://rm.sandbox.com:8088/proxy/application_1544594647974_0003/, direction: OUT via explicit rule: YARNUI/yarn/outbound/apps/history to URL: https://knox.sandbox.com:8443/gateway/default/yarn/proxy/application_1544594647974_0003
2018-12-14 03:51:26,969 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: logs, direction: OUT via implicit rule: YARNUI/yarn/outbound/logs to URL: https://knox.sandbox.com:8443/gateway/default/yarn/logs
2018-12-14 03:51:26,970 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: http://datanode03.sandbox.com:8042/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive, direction: OUT via explicit rule: YARNUI/yarn/outbound/node/containerlogs to URL: https://knox.sandbox.com:8443/gateway/default/yarn/nodemanager/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive?host=datanode03.sandbox.com&port=8042
2018-12-14 03:51:26,970 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: /cluster/app, direction: OUT via explicit rule: YARNUI/yarn/outbound/apps/app to URL: https://knox.sandbox.com:8443/gateway/default/yarn/cluster/app
2018-12-14 03:52:47,645 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /yarn/nodemanager/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive
2018-12-14 03:52:47,650 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://knox.sandbox.com:8443/gateway/default/yarn/nodemanager/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive?host=datanode03.sandbox.com&port=8042, direction: IN via implicit rule: YARNUI/yarn/inbound/node/containerlogs to URL: http://datanode03.sandbox.com:8042/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive
2018-12-14 03:52:47,650 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://knox.sandbox.com:8443/gateway/default/yarn/cluster/app/application_1544594647974_0003, direction: IN via implicit rule: YARNUI/yarn/inbound/cluster to URL: http://rm.sandbox.com:8088/cluster/app/application_1544594647974_0003
2018-12-14 03:52:47,650 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(120)) - Dispatch request: GET http://datanode03.sandbox.com:8042/node/containerlogs/container_e212_1544594647974_0003_01_000001/hive?doAs=myuser
2018-12-14 03:52:47,666 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(133)) - Dispatch response status: 307
2018-12-14 03:52:47,667 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit character set UTF-8 for entity of type text/plain
2018-12-14 03:52:47,667 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound response entity content type: text/plain; charset=utf-8

Re: knox with yarnui - colon (:) character becomes %3A

risdenk — Sat, 15 Dec 2018 03:49:43 GMT

This is most likely fixed in Apache Knox 1.2.0 (KNOX-1207) and should be fixed in HDP 3.1 just released.

Re: knox with yarnui - colon (:) character becomes %3A

manishsingh2k — Sat, 15 Dec 2018 06:40:01 GMT

Thank You Kevin. Unfortunately, an upgrade does not seem to be in the cards as of now. Would downloading the latest service directories containing the latest service and rewrite xml be able to fix it instead?

Re: knox with yarnui - colon (:) character becomes %3A

risdenk — Sat, 22 Dec 2018 03:10:15 GMT

Potentially but no guarantees. You would lose those changes on upgrades and would be up to you to keep them in sync. I know it is something that will not be supported but you seem to acknowledge that in the question with "Though KNOX with YARNUI is not officially supported"

Re: knox with yarnui - colon (:) character becomes %3A

manishsingh2k — Sat, 22 Dec 2018 05:34:53 GMT

@Kevin Risden Thank You.

question Re: knox with yarnui - colon (:) character becomes %3A in Archives of Support Questions (Read Only)

knox with yarnui - colon (:) character becomes %3A

Cannot get container logs. Invalid nodeId: datanode03.sandbox.com%3A45454

Re: knox with yarnui - colon (:) character becomes %3A

Re: knox with yarnui - colon (:) character becomes %3A

Re: knox with yarnui - colon (:) character becomes %3A

Re: knox with yarnui - colon (:) character becomes %3A