question Re: Knox - wrong mapping? help will be appriciated in Support Questions

Knox - wrong mapping? help will be appriciated

atokman — Mon, 19 Aug 2019 07:59:46 GMT

Hello,

I'm running a BigInsights cluster and I have Knox configured with LDAP Authentication and Authorization.

I've noticed that when I go to the ResourceManager UI using the Yarn quick link through Ambari UI and click on the applicationMaster link of a running job, the page doesn't look so good (like a rendering issue) and it states the following at the top:

"This page will not function without javascript enabled. Please enable javascript on your browser."

Javascript is enabled on the browser. I actually saw this in another place which turned out to be mapped wrong, so it seems to be a sympton and not the problem.

The thing is that when I try to view the running application details (using the internal link in the application manager) I get a server error, which seems like a mapping issue.

When clicking on the JobID:

Using direct link with port 8088 works well.

This is the Knox mapping I'm using:

<service> <role>NAMENODE</role> <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url> </service>

<service> <role>JOBTRACKER</role> <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url> </service>

<service> <role>WEBHDFS</role> <url>http://{{namenode_host}}:50070/webhdfs</url> </service>

<service> <role>WEBHCAT</role> <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url> </service>

<service> <role>OOZIE</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service>

<service> <role>WEBHBASE</role> <url>http://{{hbase_master_host}}:60010</url> </service>

<service> <role>HIVE</role> <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> </service>

<service> <role>RESOURCEMANAGER</role> <url>http://{{rm_host}}:{{rm_port}}/ws</url> </service>

<service> <role>HDFSUI</role> <url>http://{{namenode_host}}:50070</url> </service> <service> <role>HBASEUI</role> <url>http://{{hbase_master_host}}:{{hbase_master_ui_port}}</url> </service>

<service> <role>SPARKUI</role> <url>http://{{spark_historyserver_host}}:{{spark_historyserver_ui_port}}</url> </service> <service> <role>OOZIEUI</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service> <service> <role>YARNUI</role> <url>http://<my yarn server>:8088</url> </service> <service> <role>JOBSTORYUI</role> <url>http://{{mr_historyserver_address}}</url> </service>

Any ideas?

Thanks,

Re: Knox - wrong mapping? help will be appriciated

sshimpi — Thu, 12 May 2016 12:43:10 GMT

Hi Anat Tokman

Can you let me know the version of AMbari and HDP you are using ?

Re: Knox - wrong mapping? help will be appriciated

aervits — Thu, 12 May 2016 15:57:52 GMT

He's using IBM BigInsights

Re: Knox - wrong mapping? help will be appriciated

atokman — Sat, 14 May 2016 06:38:50 GMT

Hi thanks for your quick reply, those are the versions I'm using:

# ambari-server --version 2.1.0

# hadoop version Hadoop 2.7.1-IBM-8

Just reminding, when I use the direct link to the ResourceManager (not through Knox) it's good, so it must be something with Knox, either mapping or a Java rendering issue that I'm not aware of.

One more important thing that I noticed today - When I go to the quick link from the MapReduce service I'm able to get to the history links (using Knox), so it seems like an issue with Yarn since I see it on the Resource Manager page.

Ideas are welcome.

Thanks!

Re: Knox - wrong mapping? help will be appriciated

atokman — Sat, 14 May 2016 07:04:43 GMT

One more important thing:

about the error (pic with HTTP 404) - I can see that when I manually add the gateway path to the URL in the error, it works (partially, still with Java error, but it directs to the right place)

So if this is a job URL with an error as in picture: (it launches it when I click on the Job link)

https://myhost:8443/proxy/application_1463172884925_0003/mapreduce/job/job_1463172884925_0003

adding "/gateway/yarnui/yarn" before proxy brings me to the right place.

What am I missing?

Re: Knox - wrong mapping? help will be appriciated

pminovic — Sat, 14 May 2016 07:25:06 GMT

The mappings you cited are the ones in Ambari. Based on settings of respective services Ambari will replace {{...}} parameters with real hostnames, ports etc. You can inspect the real values used by Knox by checking files under /etc/knox/conf, like gateway-site.xml and topologies/*.xml files. If some settings are not right you can try to fix them, for example by replace mappings in Ambari with specific values. For example, you can replace {{rm_host}}:{{jt_rpc_port}} with my-rm-fqdn.hadoop.com:{{jt_rpc_port}}, and you can replace the port as well.

Re: Knox - wrong mapping? help will be appriciated

atokman — Tue, 17 May 2016 01:09:05 GMT

thanks, sure.. I checked the XMLs already. couldn't spot an issue.

</gateway><service> <role>YARNUI</role> <url>http://my-yarn-server.com:8088</url> </service></topology>

<role>RESOURCEMANAGER</role> <url>http://my-rm-server.com:8088/ws</url> </service>

</gateway><service> <role>JOBSTORYUI</role> <url>http://my-job-history-server.com.ibm.com:19888</url> </service></topology>

The issues seems to be around those 3, when the root issue looks like coming from the Resource Manager. I tend to believe that when the Resource Manager will act and look as it should, the other will aline as well.

I have the following authorization entries:

<provider> <role>authorization</role> <name>AclsAuthz</name> <enabled>true</enabled>

<param> <name>resourcemanager.acl</name> <value>*;MYPROJECT;*</value> </param>

<param> <name>yarnui.acl</name> <value>*;MYPROJECT;*</value> </param>

<param> <name>jobstoryui.acl</name> <value>*;MYPROJECT;*</value> </param>

...

(I have it for all the services).

I can try to remove authorization from those 3 and see if it changes anything.

Any other thoughts? do you see any issue with the mapping?

Thanks.

Re: Knox - wrong mapping? help will be appriciated

atokman — Wed, 18 May 2016 00:24:14 GMT

OK, so I found something that looks like an internal mapping error in YARNUI's rewrite.xml file:

<rule dir="OUT" name="YARNUI/yarn/inbound/jobhistory/logs" pattern="*1; url=*://*:*/jobhistory/logs/{**}"> <rewrite template="1; url=/gateway/jobstoryui/jobstory/jobhistory/logs/{**}"/> </rule>

Is it just me or the mapping is not right? this is the exact URL that doesn't generate the gateway path when clicking on the logs from the YARN webpage that should pass it to MapReduce.

It also looks like a mix of the IN-OUT syntax.

Those are some other rules which look better:

Can anyone let me know what should be the correct mapping for that one?

Thanks.

Re: Knox - wrong mapping? help will be appriciated

atokman — Wed, 18 May 2016 02:00:48 GMT

Apparently, this is a known bug... I replaced the rewrite.xml with the new one and now it's good.

http://www-01.ibm.com/support/docview.wss?uid=swg21980169

I hope it will help others!

Re: Knox - wrong mapping? help will be appriciated

amiller — Thu, 19 May 2016 02:32:01 GMT

Good find! Here's a copy of the workaround:

Replace /var/lib/knox/data/services/yarn-ui/2.7.1/rewrite.xml with the attached rewrite.xml (change ownership to knox:knox)
Restart Knox

Note that "data" might be version-specific (e.g. data-2.4.2.0-258), or you can use /usr/hdp/current/knox-server/data/ instead. The fixed rewrite.xml is attached.