HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

[#1]

Hi,

I am using HDP3.0 and ambari 2.7 blueprint. webhdfs via knox failed due to:

2018-08-21 19:26:33,035 ERROR knox.gateway (GatewayDispatchFilter.java:isDispatchAllowed(155)) - The dispatch to http://myhost.com:50070/webhdfs/v1/user was disallowed because it fails the dispatch whitelist validation. See documentation for dispatch whitelisting.

I have verified webhdfs without knox works:

curl -vvv http://myhost.com:50070/webhdfs/v1/user/?op=LISTSTATUS

Also, ambari, zeppelin and ranger UI work fine via knox.

The knox settings are:

gateway.dispatch.whitelist: DEFAULT

gateway.dispatch.whitelist.services: DATANODE,HBASEUI,HDFSUI,JOBHISTORYUI,NODEUI,RESOURCEMANAGER,WEBHBASE,WEBHDFS,YARNUI

webhdfs via knox worked for me on HDP2.6. Any idea? Appreciate any help.

[#2]

Looking at the documentation (https://knox.apache.org/books/knox-1-1-0/user-guide.html#Gateway+Server+Configuration), try removing

gateway.dispatch.whitelist: DEFAULT

property.

[#3 · Accepted solution]

None of Ambari, Zeppelin, or RangerUI are affected by this whitelisting.

Can you see the default whitelist in gateway.log?

It should say something like

Applying a derived dispatch whitelist because none is configured in gateway-site: xxxxxxx

[#4]

Thanks guys. I got the whitelist filter as mentioned by @Phil Zampino and updated it as my need. Then knox allowed my requests.

[#5]

what did you write in

gateway.dispatch.whitelist

???

[#6]

@Lian Jiang Can you explain why the default whitelist was not working for your deployment?

[#7]

The domain name used by hadoop hosts and the one used by the load balancer are different. DEFAULT setting will use the load balancer's domain to construct whitelist filter. I need to update the whitelist filter to use hadoop hosts' domain name instead. Hope this helps.

[#8]

Thank you for following up. That's what I suspected, and it's good to document it here for future reference.

[#9]

Hi @Phil Zampino

i have similar problem. finally what do you change ?

[#10]

@abbas mohammadnejad You have to explicitly set the gateway.dispatch.whitelist property in gateway-site.xml, such that the pattern will match the endpoint address.