Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Solved Go to solution

HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Contributor

Hi,

I am using HDP3.0 and ambari 2.7 blueprint. webhdfs via knox failed due to:

2018-08-21 19:26:33,035 ERROR knox.gateway (GatewayDispatchFilter.java:isDispatchAllowed(155)) - The dispatch to http://myhost.com:50070/webhdfs/v1/user was disallowed because it fails the dispatch whitelist validation. See documentation for dispatch whitelisting.

I have verified webhdfs without knox works:

curl -vvv http://myhost.com:50070/webhdfs/v1/user/?op=LISTSTATUS

Also, ambari, zeppelin and ranger UI work fine via knox.

The knox settings are:

gateway.dispatch.whitelist: DEFAULT

gateway.dispatch.whitelist.services: DATANODE,HBASEUI,HDFSUI,JOBHISTORYUI,NODEUI,RESOURCEMANAGER,WEBHBASE,WEBHDFS,YARNUI

webhdfs via knox worked for me on HDP2.6. Any idea? Appreciate any help.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Cloudera Employee

None of Ambari, Zeppelin, or RangerUI are affected by this whitelisting.

Can you see the default whitelist in gateway.log?

It should say something like

Applying a derived dispatch whitelist because none is configured in gateway-site: xxxxxxx

11 REPLIES 11

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Contributor

Looking at the documentation (https://knox.apache.org/books/knox-1-1-0/user-guide.html#Gateway+Server+Configuration), try removing

gateway.dispatch.whitelist: DEFAULT

property.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Cloudera Employee

None of Ambari, Zeppelin, or RangerUI are affected by this whitelisting.

Can you see the default whitelist in gateway.log?

It should say something like

Applying a derived dispatch whitelist because none is configured in gateway-site: xxxxxxx

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Contributor

Thanks guys. I got the whitelist filter as mentioned by @Phil Zampino and updated it as my need. Then knox allowed my requests.

Highlighted

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

New Contributor

what did you write in

gateway.dispatch.whitelist

???

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Cloudera Employee

@Lian Jiang Can you explain why the default whitelist was not working for your deployment?

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Contributor

The domain name used by hadoop hosts and the one used by the load balancer are different. DEFAULT setting will use the load balancer's domain to construct whitelist filter. I need to update the whitelist filter to use hadoop hosts' domain name instead. Hope this helps.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Cloudera Employee

Thank you for following up. That's what I suspected, and it's good to document it here for future reference.

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

New Contributor

Hi @Phil Zampino

i have similar problem. finally what do you change ?

Re: HDP3.0: knox fails to dispatch webhdfs request due to whitelist validation

Cloudera Employee
@abbas mohammadnejad You have to explicitly set the gateway.dispatch.whitelist property in gateway-site.xml, such that the pattern will match the endpoint address.
Don't have an account?
Coming from Hortonworks? Activate your account here