question Re: HDP 2.5 Ranger Knox policy only works with user in Archives of Support Questions (Read Only)

HDP 2.5 Ranger Knox policy only works with user

qiwang — Mon, 19 Aug 2019 09:01:03 GMT

HDP 2.5 secured cluster with Knox installed as gateway. Ranger policy is create for default topology and WEBHDFS service. However when the policy only contains only hr group which hr1 user is part of, I got error from gateway request

curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /gateway/default/webhdfs/v1/hr/exempt. Reason:
<pre>    Forbidden</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>


</body>
</html>

When user hr1 is included in the policy, it works fine

curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS


{"FileStatuses":{"FileStatus":[{"accessTime":1483632050751,"blockSize":134217728,"childrenNum":0,"fileId":152421,"group":"hr","length":23,"modificationTime":1483632051087,"owner":"hdfs","pathSuffix":"testfile","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"}]}}

This happens with other knox services as well like hive through knox. The group policy does work fine on other Ranger policies not for knox, like HDFS and Hive. Only the policies for Knox seems to have this particular problem.

And in HDFS custom core-site, I have

hadoop.proxyuser.knox.hosts=*
hadoop.proxyuser.knox.groups=*

Re: HDP 2.5 Ranger Knox policy only works with user

lmccay — Thu, 19 Jan 2017 02:14:01 GMT

More than likely, you do not have group lookup configured in Knox. If you check the {GATEWAY_HOME}/logs/gateway-audit.log you will likely notice an empty array "[]" for groups with the authentication entries.

The groups need to be looked up by the Knox code and made available to the Ranger Knox plugin. The plugin doesn't do its own group lookup.

HTH

Re: HDP 2.5 Ranger Knox policy only works with user

qiwang — Thu, 19 Jan 2017 04:23:03 GMT

@lmccay

As you mentioned, the log does indicate the group search is no returning the right group

17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|
17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|Groups: []

How do I config knox to do group lookup? Don't see anything in my topology related to group lookup

<topology>
            <gateway>
                <provider>
                    <role>authentication</role>
                    <name>ShiroProvider</name>
                    <enabled>true</enabled>
                    <param>
                        <name>sessionTimeout</name>
                        <value>30</value>
                    </param>
                    <param>
                        <name>main.ldapRealm</name>
                        <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> 
                    </param>


<!-- changes for AD/user sync -->

<param>
    <name>main.ldapContextFactory</name>
    <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>

<!-- main.ldapRealm.contextFactory needs to be placed before other main.ldapRealm.contextFactory* entries  -->
<param>
    <name>main.ldapRealm.contextFactory</name>
    <value>$ldapContextFactory</value>
</param>

<!-- AD url -->
<param>
    <name>main.ldapRealm.contextFactory.url</name>
    <value>ldap://qwang-kdc-ldap.field.hortonworks.com:389</value> 
</param>

<!-- system user -->
<param>
    <name>main.ldapRealm.contextFactory.systemUsername</name>
    <value>cn=admin,dc=field,dc=hortonworks,dc=com</value>
</param>

<!-- pass in the password using the alias created earlier -->
<param>
    <name>main.ldapRealm.contextFactory.systemPassword</name>
    <value>password</value>
</param>

                    <param>
                        <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                        <value>simple</value>
                    </param>
                    <param>
                        <name>urls./**</name>
                        <value>authcBasic</value> 
                    </param>


<!--  AD groups of users to allow -->
<param>
    <name>main.ldapRealm.searchBase</name>
    <value>ou=Users,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.userObjectClass</name>
    <value>person</value>
</param>
<param>
    <name>main.ldapRealm.userSearchAttributeName</name>
    <value>uid</value>
</param>


<!-- changes needed for group sync-->
<param>
    <name>main.ldapRealm.authorizationEnabled</name>
    <value>true</value>
</param>
<param>
    <name>main.ldapRealm.groupSearchBase</name>
    <value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.groupObjectClass</name>
    <value>posixgroup</value>
</param>
<param>
    <name>main.ldapRealm.groupIdAttribute</name>
    <value>cn</value>
</param>
                </provider>
                <provider>
                    <role>identity-assertion</role>
                    <name>Default</name>
                    <enabled>true</enabled>
                </provider>
                <provider>
                    <role>authorization</role>
                    <name>XASecurePDPKnox</name>
                    <enabled>true</enabled>
                </provider>
            </gateway>
            
            <service>
                <role>NAMENODE</role>
                <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url>
            </service>
            <service>
                <role>JOBTRACKER</role>
                <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
            </service>
            <service>
                <role>WEBHDFS</role>
                <url>http://{{namenode_host}}:{{namenode_http_port}}/webhdfs</url>
            </service>
            <service>
                <role>WEBHCAT</role>
                <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
            </service>
            <service>
                <role>OOZIE</role>
                <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
            </service>
            <service>
                <role>WEBHBASE</role>
                <url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
            </service>
            <service>
                <role>HIVE</role>
                <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
            </service>
            <service>
                <role>RESOURCEMANAGER</role>
                <url>http://{{rm_host}}:{{rm_port}}/ws</url>
            </service>
        </topology>

Re: HDP 2.5 Ranger Knox policy only works with user

dvillarreal — Thu, 19 Jan 2017 06:07:56 GMT

Hi @Qi Wang,

This may also help where I have answered a similar question.

https://community.hortonworks.com/questions/74501/how-knox-pass-the-user-information-to-ranger.html

Re: HDP 2.5 Ranger Knox policy only works with user

qiwang — Thu, 19 Jan 2017 07:09:06 GMT

@dvillarreal

I read your comments on

https://community.hortonworks.com/articles/38348/ranger-is-not-allowing-access-to-knox-resources-wh.html

If you look at my topology, it contains the group section just like in your post. The only difference I could think of is that I use open-ldap as the directory server rather than AD. And the values for object class and attributes are different than AD.

<param>
    <name>main.ldapRealm.authorizationEnabled</name>
    <value>true</value>
</param>
<param>
    <name>main.ldapRealm.groupSearchBase</name>
    <value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.groupObjectClass</name>
    <value>posixgroup</value>
</param>
<param>
    <name>main.ldapRealm.groupIdAttribute</name>
    <value>cn</value>
</param>

Re: HDP 2.5 Ranger Knox policy only works with user

dvillarreal — Fri, 20 Jan 2017 03:34:00 GMT

Hi @Qi Wang

This should help you to learn by example when it comes to configuring your knox groups and how it relates to your ldapsearch. See Sample 4 specifically https://cwiki.apache.org/confluence/display/KNOX/Using+Apache+Knox+with+ActiveDirectory

Hope this helps.