question HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation in Archives of Support Questions (Read Only)

HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

jiangok2006 — Wed, 22 Aug 2018 02:36:53 GMT

Hi,

I am using HDP3.0 and ambari 2.7 blueprint. webhdfs via knox failed due to:

2018-08-21 19:26:33,035 ERROR knox.gateway (GatewayDispatchFilter.java:isDispatchAllowed(155)) - The dispatch to http://myhost.com:50070/webhdfs/v1/user was disallowed because it fails the dispatch whitelist validation. See documentation for dispatch whitelisting.

I have verified webhdfs without knox works:

curl -vvv http://myhost.com:50070/webhdfs/v1/user/?op=LISTSTATUS

Also, ambari, zeppelin and ranger UI work fine via knox.

The knox settings are:

gateway.dispatch.whitelist: DEFAULT

gateway.dispatch.whitelist.services: DATANODE,HBASEUI,HDFSUI,JOBHISTORYUI,NODEUI,RESOURCEMANAGER,WEBHBASE,WEBHDFS,YARNUI

webhdfs via knox worked for me on HDP2.6. Any idea? Appreciate any help.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

smore — Thu, 23 Aug 2018 05:33:38 GMT

Looking at the documentation (https://knox.apache.org/books/knox-1-1-0/user-guide.html#Gateway+Server+Configuration), try removing

gateway.dispatch.whitelist: DEFAULT

property.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

pzampino — Thu, 23 Aug 2018 05:42:29 GMT

None of Ambari, Zeppelin, or RangerUI are affected by this whitelisting.

Can you see the default whitelist in gateway.log?

It should say something like

Applying a derived dispatch whitelist because none is configured in gateway-site: xxxxxxx

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

jiangok2006 — Fri, 24 Aug 2018 06:51:49 GMT

Thanks guys. I got the whitelist filter as mentioned by @Phil Zampino and updated it as my need. Then knox allowed my requests.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

pzampino — Fri, 24 Aug 2018 09:26:57 GMT

@Lian Jiang Can you explain why the default whitelist was not working for your deployment?

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

jiangok2006 — Sat, 25 Aug 2018 03:58:10 GMT

The domain name used by hadoop hosts and the one used by the load balancer are different. DEFAULT setting will use the load balancer's domain to construct whitelist filter. I need to update the whitelist filter to use hadoop hosts' domain name instead. Hope this helps.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

pzampino — Sat, 25 Aug 2018 04:00:51 GMT

Thank you for following up. That's what I suspected, and it's good to document it here for future reference.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

abbas_kurdistan — Wed, 30 Jan 2019 16:44:18 GMT

Hi @Phil Zampino

i have similar problem. finally what do you change ?

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

pzampino — Thu, 31 Jan 2019 01:41:59 GMT

@abbas mohammadnejad You have to explicitly set the gateway.dispatch.whitelist property in gateway-site.xml, such that the pattern will match the endpoint address.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

gorkazar7 — Mon, 20 May 2019 15:49:33 GMT

what did you write in

gateway.dispatch.whitelist

???

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

gorkazar7 — Mon, 20 May 2019 22:58:42 GMT

I have fill that property with

gateway.dispatch.whitelist --> ^https?:\/\/(my.domain.com|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$

But i have all the time the same error

ERROR knox.gateway (GatewayDispatchFilter.java:isDispatchAllowed(155)) - The dispatch to http://my.domain.com:50070/webhdfs/v1/ was disallowed because it fails the dispatch whitelist validation. See documentation for dispatch whitelisting.

Thanks

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

ssummers — Thu, 13 Jun 2019 21:25:52 GMT

I ran into a similar issue to this as well. It looks like when it derives the RegEX based off the DEFAULT. We found that it used the URL from the Provider we used setting up the SSO client we had typed in just the Hostname for the Knox Server. It then resulted in the following Default white list:

INFO  knox.gateway (WhitelistUtils.java:getDispatchWhitelist(63)) - Applying a derived dispatch whitelist because none is configured in gateway-site: ^/.*$;^https?://ambari30l:[0-9]+/?.*$

After redoing the SSO setup with the FQDN for the host it resolved the "DEFAULT" Lookup and was able to find the hosts properly.

INFO  knox.gateway (WhitelistUtils.java:getDispatchWhitelist(63)) - Applying a derived dispatch whitelist because none is configured in gateway-site: ^/.*$;^https?://ambari30l.mydomain.com:[0-9]+/?.*$

Hopefully this help others in troubleshooting this issue in the future.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

khannanitish00 — Tue, 03 Jan 2023 20:07:19 GMT

Ssumers,

Can you please share the value you added in gateway.dispatch.whitelist?