Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

WebHDFS via Knox: Access Denied

WebHDFS via Knox: Access Denied

New Contributor

WebHDFS via Knox: Access Denied

Hi,

Setup: Ambari 2.7, HDP 3.0.

I use Kerberos for authentication in WebHDFS via http. And I use Ranger for authentication.

So following command works as expected:

curl -v --negotiate -u : -X GET "http://mywebhdfs:50070/webhdfs/v1/tmp?op=LISTSTATUS"

So i setup and configuered Knox.

My Advanced Topology looks like this:

<gateway>

<provider>
<role>authentication</role>
<name>HadoopAuth</name>
<enabled>true</enabled>
<param>
<name>config.prefix</name>
<value>hadoop.auth.config</value>
</param>
<param>
<name>hadoop.auth.config.signature.secret</name>
<value>knox-signature-secret</value>
</param>
<param>
<name>hadoop.auth.config.type</name>
<value>kerberos</value>
</param>
<param>
<name>hadoop.auth.config.simple.anonymous.allowed</name>
<value>false</value>
</param>
<param>
<name>hadoop.auth.config.token.validity</name>
<value>1800</value>
</param>
<param>
<name>hadoop.auth.config.cookie.domain</name>
<value>*my.domain.com*</value>
</param>
<param>
<name>hadoop.auth.config.cookie.path</name>
<value>gateway/default</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.principal</name>
<value>HTTP/*myknoxhost*@*REALM*</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.keytab</name>
<value>/etc/security/keytabs/spnego.service.keytab</value>
</param>
<param>
<name>hadoop.auth.config.kerberos.name.rules</name>
<value>DEFAULT</value>
</param>
</provider>

<provider>
<role>identity-assertion</role>
<name>HadoopGroupProvider</name>
<enabled>true</enabled>
</provider>

<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>

</gateway>

... + services ...

I I try to access the same folder with the same Kerberos ticket via

curl -v --negotiate -u : -X GET "https://*myknoxhost*:8443/gateway/default/webhdfs/v1/tmp?op=LISTSTATUS"

I receive error 403 permission denied:

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /gateway/default/webhdfs/v1/tmp. Reason:
<pre> Forbidden</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

However, there is nothing to see in the Ranger Audits about this. Auditing is enabled, as per default.
In the Knox Gateway Logs I can see, that there was the invocation with 403:

18/07/26 13:12:19 ||2463f7a5-0158-46dd-ac61-1a2c5688fb76|audit|myip|WEBHDFS|myuser|||authentication|uri|/gateway/default/webhdfs/v1/tmp?op=LISTSTATUS|success|
18/07/26 13:12:19 ||2463f7a5-0158-46dd-ac61-1a2c5688fb76|audit|myip|WEBHDFS|myuser|||identity-mapping|principal|myuser|success|Groups: [my, list, of, groups]
18/07/26 13:12:19 |||audit|10.1.120.101|WEBHDFS|myuser|||access|uri|/gateway/default/webhdfs/v1/tmp?op=LISTSTATUS|success|Response status: 403

Finally I set the knox Log level higher and i can recieve this log information:

84414-temp2.pngtemp2.png(136.8 kB)

I cant find any reason why access is denied. Is this possibly a bug? The user definitely has access to the resource in WebHDFs.

Best

Luke

6 REPLIES 6

Re: WebHDFS via Knox: Access Denied

Hi @Luke Luke
Have you attempted this also with SwitchCase in effect?
Eg

<role>identity-assertion</role>
<name>HadoopGroupProvider</name>

Into

<role>identity-assertion</role>
<name>SwitchCase</name> 

Re: WebHDFS via Knox: Access Denied

New Contributor

@Jonathan Sneep

Thanks. I tried it, but it didn't help.

The strange thing is that the request does not even show up in die audit logs of ranger. (the request via webhdfs directly shows up) So maybe there is an issue with the RangerPDPKnoxfilter...? But i cant debug it further :/

Re: WebHDFS via Knox: Access Denied

Frustrating...
https://community.hortonworks.com/content/supportkb/150159/ranger-is-unable-to-access-knox-services....
I forgot to include mention of the group.principal.case and principal.case settings as seen on the above KB, it may be worth another shot with those included.
Also, on core-site settings, have you checked the proxyuser settings? You could wildcard these briefly to see if it has an effect.

hadoop.proxyuser.knox.groups
hadoop.proxyuser.knox.hosts

Re: WebHDFS via Knox: Access Denied

New Contributor

indeed. Still frustrating.

I already configured the proxyusers to "*".

Also your solution with the SwitchCase didn't help. I get exactly the same error. And btw: In my LDAP all users and groups are in lowercase.

The most strange thing at all is that there is absolutely nothing about it in the Ranger Audit Logs i can view from the Ranger UI. It seems that I never made this GET Request to HDFS using Knox...

Any more advice anyone? Spent already 2,5 days with this problem :/

Highlighted

Re: WebHDFS via Knox: Access Denied

Super Collaborator
@Luke Luke

Can you show us how the knox ranger policy is defined from Ranger admin UI? or from policy cache on knox host.

Not sure why audit is not written in your case but this is for sure Ranger plugin acting here.

Re: WebHDFS via Knox: Access Denied

New Contributor

Hi @Luke Luke

Did you found the solution to this issue ?

We have same problem in our env.

Thanks,

,

Hi @Luke Luke

Did you find the solution for this problem ?

We are facing same issue in our environment.

Thanks,

Don't have an account?
Coming from Hortonworks? Activate your account here