Recommendation for proxying Hadoop services without built-in Knox support

[#1]

Knox 0.6.0 has built-in support for these 7 services:

• WebHDFS
• WebHCat
• Oozie
• HBase
• Hive
• Yarn
• Storm

Is there a recommended approach to expose other services from the gateway host? Particularly web UIs, such as Ambari & Ranger.

[#2]

@Kevin Minder

[#3]

I suppose you can use haproxy for example. However if you have kerberos and spnego you would need to add the proxy tickets similar to the oozie ha setup described here in the cloudera doc ( I would use ours if we would actually describe that ) http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_sg_oozie_ha_ker...

[#4]

I figured something like haproxy or nginx would work. Preferably looking for an example config, or if anyone has extended Knox with a custom provider then even better.

[#5]

We would certainly recommend the use of Knox's extensibility models to cover any components without coverage before we get there ourselves. There have been several developed in the community already such as Falcon that we don't yet officially support. The same goes for UI coverage where the community has added coverage for things like the HDFS and YARN UIs among others. The Knox Developer's Guide is a great resource that the community has used to help them jump start these efforts. Of course looking at the implementation of the existing integrations is a great place to start as well.

[#6]

The openweathermap example in the Knox Dev Guide looks great as a reference for extending Knox yourself. Do you know where some existing community extensions, like the Falcon or NN/RM UIs, can be found? I checked the Hortonworks Gallery with no luck.

[#7 · Accepted solution]

These extensions are committed to the Apache Knox repo itself. They all use the config driven extension model so you need to look in the gateway-service-definitions module. In particular look in this directory. Now that you mention the openweathermap example, I need to update that to the new configuration based model at least as a comparison to the code based extension. The developers guide does briefly cover the config based extension.

[#8]

I also recently created a blog related to this. http://kminder.github.io/knox/2015/11/16/adding-a-service-to-knox.html

[#9]

@Kevin Minder moving to the best answer.

[#10]

For quick reference, here's an example of adding Oozie UI to HDP 2.4 Sandbox:

1. start Sandbox and make sure all non-maintenance services are running

2. add service definition:

git clone https://git-wip-us.apache.org/repos/asf/knox.git
cp -R knox/gateway-service-definitions/src/main/resources/services/oozieui /var/lib/knox/data-2.4.0.0-169/services/
chown -R knox:knox /var/lib/knox/data-2.4.0.0-169/services/oozieui

3. add OOZIEUI service to default.xml topology (Ambari > Knox > Configs > Advanced topology)

<service>
    <role>OOZIEUI</role>
    <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
  </service>

4. start (or restart) Knox & Demo LDAP (using Ambari)

5. visit https://localhost:8443/gateway/default/oozie/