Knox - wrong mapping? help will be appriciated

[#1]

Hello,

I'm running a BigInsights cluster and I have Knox configured with LDAP Authentication and Authorization.

I've noticed that when I go to the ResourceManager UI using the Yarn quick link through Ambari UI and click on the applicationMaster link of a running job, the page doesn't look so good (like a rendering issue) and it states the following at the top:

"This page will not function without javascript enabled. Please enable javascript on your browser."

Javascript is enabled on the browser. I actually saw this in another place which turned out to be mapped wrong, so it seems to be a sympton and not the problem.

The thing is that when I try to view the running application details (using the internal link in the application manager) I get a server error, which seems like a mapping issue.

When clicking on the JobID:

Using direct link with port 8088 works well.

This is the Knox mapping I'm using:

<service> <role>NAMENODE</role> <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url> </service>

<service> <role>JOBTRACKER</role> <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url> </service>

<service> <role>WEBHDFS</role> <url>http://{{namenode_host}}:50070/webhdfs</url> </service>

<service> <role>WEBHCAT</role> <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url> </service>

<service> <role>OOZIE</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service>

<service> <role>WEBHBASE</role> <url>http://{{hbase_master_host}}:60010</url> </service>

<service> <role>HIVE</role> <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> </service>

<service> <role>RESOURCEMANAGER</role> <url>http://{{rm_host}}:{{rm_port}}/ws</url> </service>

<service> <role>HDFSUI</role> <url>http://{{namenode_host}}:50070</url> </service> <service> <role>HBASEUI</role> <url>http://{{hbase_master_host}}:{{hbase_master_ui_port}}</url> </service>

<service> <role>SOLR</role> <url>http://{{solr_host}}:{{solr_port}}/solr</url> </service>

<service> <role>SPARKUI</role> <url>http://{{spark_historyserver_host}}:{{spark_historyserver_ui_port}}</url> </service> <service> <role>OOZIEUI</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service> <service> <role>YARNUI</role> <url>http://<my yarn server>:8088</url> </service> <service> <role>JOBSTORYUI</role> <url>http://{{mr_historyserver_address}}</url> </service>

Any ideas?

Thanks,

AT

[#2]

Hi Anat Tokman

Can you let me know the version of AMbari and HDP you are using ?

[#3]

He's using IBM BigInsights

[#4]

Hi thanks for your quick reply, those are the versions I'm using:

# ambari-server --version 2.1.0

# hadoop version Hadoop 2.7.1-IBM-8

Just reminding, when I use the direct link to the ResourceManager (not through Knox) it's good, so it must be something with Knox, either mapping or a Java rendering issue that I'm not aware of.

One more important thing that I noticed today - When I go to the quick link from the MapReduce service I'm able to get to the history links (using Knox), so it seems like an issue with Yarn since I see it on the Resource Manager page.

Ideas are welcome.

Thanks!

[#5]

One more important thing:

about the error (pic with HTTP 404) - I can see that when I manually add the gateway path to the URL in the error, it works (partially, still with Java error, but it directs to the right place)

So if this is a job URL with an error as in picture: (it launches it when I click on the Job link)

https://myhost:8443/proxy/application_1463172884925_0003/mapreduce/job/job_1463172884925_0003

adding "/gateway/yarnui/yarn" before proxy brings me to the right place.

What am I missing?

[#6]

The mappings you cited are the ones in Ambari. Based on settings of respective services Ambari will replace {{...}} parameters with real hostnames, ports etc. You can inspect the real values used by Knox by checking files under /etc/knox/conf, like gateway-site.xml and topologies/*.xml files. If some settings are not right you can try to fix them, for example by replace mappings in Ambari with specific values. For example, you can replace {{rm_host}}:{{jt_rpc_port}} with my-rm-fqdn.hadoop.com:{{jt_rpc_port}}, and you can replace the port as well.

[#7]

thanks, sure.. I checked the XMLs already. couldn't spot an issue.

</gateway><service> <role>YARNUI</role> <url>http://my-yarn-server.com:8088</url> </service></topology>

<role>RESOURCEMANAGER</role> <url>http://my-rm-server.com:8088/ws</url> </service>

</gateway><service> <role>JOBSTORYUI</role> <url>http://my-job-history-server.com.ibm.com:19888</url> </service></topology>

The issues seems to be around those 3, when the root issue looks like coming from the Resource Manager. I tend to believe that when the Resource Manager will act and look as it should, the other will aline as well.

I have the following authorization entries:

<provider> <role>authorization</role> <name>AclsAuthz</name> <enabled>true</enabled>

<param> <name>resourcemanager.acl</name> <value>*;MYPROJECT;*</value> </param>

<param> <name>yarnui.acl</name> <value>*;MYPROJECT;*</value> </param>

<param> <name>jobstoryui.acl</name> <value>*;MYPROJECT;*</value> </param>

...

...

(I have it for all the services).

I can try to remove authorization from those 3 and see if it changes anything.

Any other thoughts? do you see any issue with the mapping?

Thanks.

[#8]

OK, so I found something that looks like an internal mapping error in YARNUI's rewrite.xml file:

<rule dir="OUT" name="YARNUI/yarn/inbound/jobhistory/logs" pattern="*1; url=*://*:*/jobhistory/logs/{**}"> <rewrite template="1; url=/gateway/jobstoryui/jobstory/jobhistory/logs/{**}"/> </rule>

Is it just me or the mapping is not right? this is the exact URL that doesn't generate the gateway path when clicking on the logs from the YARN webpage that should pass it to MapReduce.

It also looks like a mix of the IN-OUT syntax.

Those are some other rules which look better:

<rule dir="IN" name="YARNUI/yarn/inbound/resourcemanager/home" pattern="*://*:*/**/yarn/resourcemanager?{scheme}?{host}?{port}"> <rewrite template="{scheme}://{host}:{port}/cluster"/> </rule> <rule dir="IN" name="YARNUI/yarn/inbound/logs" pattern="*://*:*/**/yarn/logs?{scheme}?{host}?{port}?{**}"> <rewrite template="{scheme}://{host}:{port}/logs/?{**}"/> </rule> <rule dir="IN" name="YARNUI/yarn/inbound/jobhistory/job" pattern="*://*:*/**/yarn/jobhistory/job/{**}?{scheme}?{host}?{port}"> <rewrite template="{scheme}://{host}:{port}/jobhistory/job/{**}"/> </rule> <rule dir="IN" name="YARNUI/yarn/inbound/sparkhistory/job" pattern="*://*:*/**/yarn/history/{**}?{**}?{scheme}?{host}?{port}"> <rewrite template="{scheme}://{host}:{port}/history/{**}?{**}"/> </rule>

<rule dir="OUT" name="YARNUI/yarn/outbound/headers/logs/location"> <match pattern="{scheme}://{host}:{port}/logs/?{**}"/> <rewrite template="{$frontend[url]}/yarn/logs?{scheme}?host={$hostmap(host)}?{port}?{**}"/> </rule> <rule dir="OUT" name="YARNUI/yarn/outbound/logs/files" pattern="/logs/{**}"> <rewrite template="{$frontend[url]}/yarn/logs/{**}"/> </rule> <rule dir="OUT" name="YARNUI/yarn/outbound/static" pattern="/static/{**}"> <rewrite template="{$frontend[url]}/yarn/static/{**}"/>

Can anyone let me know what should be the correct mapping for that one?

Thanks.

[#9 · Accepted solution]

Apparently, this is a known bug... I replaced the rewrite.xml with the new one and now it's good.

http://www-01.ibm.com/support/docview.wss?uid=swg21980169

I hope it will help others!

[#10]

Good find! Here's a copy of the workaround:

1. Replace /var/lib/knox/data/services/yarn-ui/2.7.1/rewrite.xml with the attached rewrite.xml (change ownership to knox:knox)
2. Restart Knox

Note that "data" might be version-specific (e.g. data-2.4.2.0-258), or you can use /usr/hdp/current/knox-server/data/ instead. The fixed rewrite.xml is attached.