Knox UI Mapping Problem - ResourceManager Web UI

[#1]

I would like to map the ResourceManager Web UI with Knox.

I've set up the topology like:

<service> <role>YARNUI</role> <url>http://{{rm_host}}:8088/</url> </service>
<service> <role>HDFSUI</role> <url>http://{{namenode_host}}:50070/</url> </service>

But the browser can't load the .css and .javascript from RM UI. "This page will not function without javascript enabled. Please enable javascript on your browser."

I'm using Google Chrome but there are the same problems with Internet Explorer.

JavaScript is working and RM UI directly is working as well. Developer Tool from Chrome: Failed to load resource: the server responded with a status of 404 (Not Found)

Same Problems with NameNode UI.

Can somebody help me please, or are there known problems?

Thanks, Andreas

[#2]

You can try and add a / in your browser when you make the request.

Otherwise, look at the github code and replace the xmls for the services you have issues with the ones on github, for example replace /usr/hdp/current/knox-server/data/services/hdfsui/2.7.0/ with https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/...

[#3]

Thanks for your fast answer. I already tried it with a / at the end, I know this is a known bug in Oozie UI mapping..

I also replaced the service.xml and rewrite.xml with both github .xml's. After a Knox restart, the problems are still the same...

My URLs:

https://<our-knox-host>:8443/gateway/default/yarn/

https://<our-knox-host>:8443/gateway/default/hdfs/

Example call for Oozie (it's working without any problems):

https://<our-knox-host>:8443/gateway/default/oozie/

[#4]

NameNode UI:

[#5]

What Knox / HDP version do you have?

It works fine on my HDP 2.5.3 Knox.

Have you also defined WEBHDFS and RESOURCEMANAGER in your topology?

Any errors in gateway.log and do you see the same 404s in gateway-audit.log?

[#6]

I'm using HDP 2.5.3 as well, with Knox version 0.9.0.2.5

Yes, I have also defined WEBHDFS and RESOURCEMANAGER like the default config in my topology, it's working without any problems:

<service> <role>NAMENODE</role> <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url> </service>
<service> <role>JOBTRACKER</role> <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url> </service> 
<service> <role>WEBHDFS</role> {{webhdfs_service_urls}} </service> 
<service> <role>WEBHCAT</role> <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url> </service>
<service> <role>OOZIE</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service> 
<service> <role>WEBHBASE</role> <url>http://{{hbase_master_host}}:{{hbase_master_port}}</url> </service> 
<service> <role>HIVE</role> <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> </service>
<service> <role>RESOURCEMANAGER</role> <url>http://{{rm_host}}:{{rm_port}}/ws</url> </service>

I see the same 404s in the gateway-audit.log for example:

access|uri|/gateway/default/hdfs/static/bootstrap-3.0.2/js/bootstrap.min.js|success|Response status: 404

[#7]

It's really strange @Andreas Schild I've replicated your configuration on my cluster and still works fine, IE or Firefox.

Let's try the following troubleshooting steps.

1) First make sure the topology file has actually been resolved with the correct values:

cat /etc/knox/conf/topologies/default.xml

2) Get from it the URL for HDFSUI, append /static/bootstrap-3.0.2/js/bootstrap.min.js to it and try it out in a curl, should be like this:

curl -i http://namenode:50070/static/bootstrap-3.0.2/js/bootstrap.min.js

3) I'd also want to see the full line the gateway-audit.log, mine shows additional information (like the service name) and also the dispatch:

17/02/16 17:15:53 ||c5f07e39-ab12-4a71-9283-f37a35128419|audit|HDFSUI|guest|||dispatch|uri|http://namenode:50070/static/bootstrap-3.0.2/js/bootstrap.min.js?doAs=guest|success|Response status: 200

17/02/16 17:15:53 ||c5f07e39-ab12-4a71-9283-f37a35128419|audit|HDFSUI|guest|||access|uri|/gateway/default/hdfs/static/bootstrap-3.0.2/js/bootstrap.min.js|success|Response status: 200

4) Check under /usr/hdp/current/knox-server/data/services/ if you have all services, I have:

ls -l /usr/hdp/current/knox-server/data/services/
total 0
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 ambari
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 ambariui
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 falcon
drwxr-xr-x. 3 knox knox 19 Feb  7 09:41 hbase
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 hbaseui
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 hdfsui
drwxr-xr-x. 3 knox knox 19 Feb  7 09:41 hive
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 jobhistoryui
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 oozie
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 oozieui
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 ranger
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 rangerui
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 sparkhistoryui
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 storm
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 storm-logviewer
drwxr-xr-x. 3 knox knox 19 Feb  7 09:41 webhcat
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 webhdfs
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 yarn-rm
drwxr-xr-x. 3 knox knox 18 Feb  7 09:41 yarnui

5) Stop Knox and delete/move /usr/hdp/2.5.3.0-37/knox/bin/../data/deployments/default*

[#8]

Hi @Alexandru Anghel,

at first, thank you very much for your help!! We have a hardware firewall with DMZ enabled in our cluster. It blocks all ports except 8443 for knox. Do you think this might be a problem?

I checked your troubleshooting steps:

1.) Yes, the hostnames and ports are correctly resolved in default.xml

2.) curl on the namenode is working fine

curl -i http://our_namenode_host:50070/static/bootstrap-3.0.2/js/bootstrap.min.js 

HTTP/1.1 200 OK X-FRAME-OPTIONS: SAMEORIGIN Accept-Ranges: bytes Content-Type: application/x-javascript Content-Length: 27748 Last-Modified: Tue, 29 Nov 2016 20:33:00 GMT Server: Jetty(6.1.26.hwx)

3.) Checked the gateway-audit.log

17/02/17 06:49:42 ||7daf998a-1783-41a3-b3bd-008f588227b3|audit|HDFSUI||||access|uri|/gateway/default/hdfs/|unavailable|Request method: GET
17/02/17 06:49:42 ||7daf998a-1783-41a3-b3bd-008f588227b3|audit|HDFSUI||||access|uri|/gateway/default/hdfs/|success|Response status: 401
17/02/17 06:49:42 ||669c7b17-c633-4dfe-adac-68b88a5c0217|audit|HDFSUI||||access|uri|/gateway/default/hdfs/|unavailable|Request method: GET
17/02/17 06:49:44 ||669c7b17-c633-4dfe-adac-68b88a5c0217|audit|HDFSUI|<my_user>|||authentication|uri|/gateway/default/hdfs/|success|

17/02/17 06:49:45 ||2883703f-494a-4d99-8151-cf27b13e4924|audit|KNOX|<my_user>|||access|uri|/gateway/default/hdfs/static/dataTables.bootstrap.css|success|Response status: 404
17/02/17 06:49:45 ||d0b6cc4d-ed05-4b9a-beae-9b267d4dfc39|audit|KNOX||||access|uri|/gateway/default/hdfs/static/dust-helpers-1.1.1.min.js|unavailable|Request method: GET
17/02/17 06:49:45 ||d0b6cc4d-ed05-4b9a-beae-9b267d4dfc39|audit|KNOX|<my_user>|||authentication|uri|/gateway/default/hdfs/static/bootstrap-3.0.2/js/bootstrap.min.js|success|
17/02/17 06:49:45 ||d0b6cc4d-ed05-4b9a-beae-9b267d4dfc39|audit|KNOX|

<my_user>

|||authentication|uri|/gateway/default/hdfs/static/bootstrap-3.0.2/js/bootstrap.min.js|success|Groups: [<my_AD_groups>]

4.) I checked

<user>@<knox_node>:/usr/hdp/current/knox-server/data/services> ll
total 76
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 ambari
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 ambariui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 falcon
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 hbase
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 hbaseui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 hdfsui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 hive
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 jobhistoryui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 oozie
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 oozieui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 ranger
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 rangerui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 sparkhistoryui
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 storm
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 storm-logviewer
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 webhcat
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 webhdfs
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 yarn-rm
drwxr-xr-x 3 knox knox 4096 Dec 21 12:01 yarnui

5.) I stopped knox, moved the /deployments/default* and started knox again, but still the same problems.

Thanks,

Andreas

[#9]

I did the curl command from 2.) from one of the cluster nodes (internal). I tested it now from my client pc with the following command:

C:\01_Data\curl\src>curl curl -i -k -su <my_user> -X GET "https://<our_knox_host>:8443/gateway/default/static/bootstrap-3.0.2/js/bootstrap.min.js"
Enter host password for user '<my_user>':
HTTP/1.1 404 Not Found
Date: Fri, 17 Feb 2017 07:01:45 GMT
Set-Cookie: JSESSIONID=174cdd7p72bdupmc98his5fcr;Path=/gateway/default;Secure;HttpOnly
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Thu, 16-Feb-2017 07:01:46 GMT
Content-Type: text/html; charset=ISO-8859-1
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 342
Server: Jetty(9.2.15.v20160210)

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /gateway/default/static/bootstrap-3.0.2/js/bootstrap.min.js. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>

The Oozie UI is the only one which is working, Oozie is installed on the same node like the our knox gateway. Maybe there is a problem by accessing the NameNode host or Ranger host from the knox gateway (knox host).

[#10]

Could be network although I would have expected a different error. Maybe there are other errors in the logs pointing to a connectivity issue?

I still don't see any dispatch in your log, that's the request Knox would make to the NameNode, do you see it in your audit log like in my example?

The second check, the curl -i http://our_namenode_host:50070/static/bootstrap-3.0.2/js/bootstrap.min.js, did you do it from the knox host?

I've seen weird behaviours in the past when HTTP proxies are configured, if you have one on the knox box, can you make sure the namenode host is on the NO_PROXY / ignore list?

With a proxy, your shell session / user / curl might be allowed to correctly make the request but knox would go via the proxy?