Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP 2.5 Ranger Knox policy only works with user

avatar
Master Collaborator

HDP 2.5 secured cluster with Knox installed as gateway. Ranger policy is create for default topology and WEBHDFS service. However when the policy only contains only hr group which hr1 user is part of, I got error from gateway request

curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /gateway/default/webhdfs/v1/hr/exempt. Reason:
<pre>    Forbidden</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>


</body>
</html>

11497-screen-shot-2017-01-18-at-102746-am.png

When user hr1 is included in the policy, it works fine

curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS


{"FileStatuses":{"FileStatus":[{"accessTime":1483632050751,"blockSize":134217728,"childrenNum":0,"fileId":152421,"group":"hr","length":23,"modificationTime":1483632051087,"owner":"hdfs","pathSuffix":"testfile","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"}]}}

11498-screen-shot-2017-01-18-at-102703-am.png

This happens with other knox services as well like hive through knox. The group policy does work fine on other Ranger policies not for knox, like HDFS and Hive. Only the policies for Knox seems to have this particular problem.

And in HDFS custom core-site, I have

hadoop.proxyuser.knox.hosts=*
hadoop.proxyuser.knox.groups=*
1 ACCEPTED SOLUTION

avatar

Hi @Qi Wang

This should help you to learn by example when it comes to configuring your knox groups and how it relates to your ldapsearch. See Sample 4 specifically https://cwiki.apache.org/confluence/display/KNOX/Using+Apache+Knox+with+ActiveDirectory

Hope this helps.

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

More than likely, you do not have group lookup configured in Knox. If you check the {GATEWAY_HOME}/logs/gateway-audit.log you will likely notice an empty array "[]" for groups with the authentication entries.

The groups need to be looked up by the Knox code and made available to the Ranger Knox plugin. The plugin doesn't do its own group lookup.

HTH

avatar
Master Collaborator

@lmccay

As you mentioned, the log does indicate the group search is no returning the right group

17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|
17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|Groups: []

How do I config knox to do group lookup? Don't see anything in my topology related to group lookup

<topology>
            <gateway>
                <provider>
                    <role>authentication</role>
                    <name>ShiroProvider</name>
                    <enabled>true</enabled>
                    <param>
                        <name>sessionTimeout</name>
                        <value>30</value>
                    </param>
                    <param>
                        <name>main.ldapRealm</name>
                        <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> 
                    </param>


<!-- changes for AD/user sync -->

<param>
    <name>main.ldapContextFactory</name>
    <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>

<!-- main.ldapRealm.contextFactory needs to be placed before other main.ldapRealm.contextFactory* entries  -->
<param>
    <name>main.ldapRealm.contextFactory</name>
    <value>$ldapContextFactory</value>
</param>

<!-- AD url -->
<param>
    <name>main.ldapRealm.contextFactory.url</name>
    <value>ldap://qwang-kdc-ldap.field.hortonworks.com:389</value> 
</param>

<!-- system user -->
<param>
    <name>main.ldapRealm.contextFactory.systemUsername</name>
    <value>cn=admin,dc=field,dc=hortonworks,dc=com</value>
</param>

<!-- pass in the password using the alias created earlier -->
<param>
    <name>main.ldapRealm.contextFactory.systemPassword</name>
    <value>password</value>
</param>

                    <param>
                        <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                        <value>simple</value>
                    </param>
                    <param>
                        <name>urls./**</name>
                        <value>authcBasic</value> 
                    </param>


<!--  AD groups of users to allow -->
<param>
    <name>main.ldapRealm.searchBase</name>
    <value>ou=Users,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.userObjectClass</name>
    <value>person</value>
</param>
<param>
    <name>main.ldapRealm.userSearchAttributeName</name>
    <value>uid</value>
</param>


<!-- changes needed for group sync-->
<param>
    <name>main.ldapRealm.authorizationEnabled</name>
    <value>true</value>
</param>
<param>
    <name>main.ldapRealm.groupSearchBase</name>
    <value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.groupObjectClass</name>
    <value>posixgroup</value>
</param>
<param>
    <name>main.ldapRealm.groupIdAttribute</name>
    <value>cn</value>
</param>
                </provider>
                <provider>
                    <role>identity-assertion</role>
                    <name>Default</name>
                    <enabled>true</enabled>
                </provider>
                <provider>
                    <role>authorization</role>
                    <name>XASecurePDPKnox</name>
                    <enabled>true</enabled>
                </provider>
            </gateway>
            
            <service>
                <role>NAMENODE</role>
                <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url>
            </service>
            <service>
                <role>JOBTRACKER</role>
                <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
            </service>
            <service>
                <role>WEBHDFS</role>
                <url>http://{{namenode_host}}:{{namenode_http_port}}/webhdfs</url>
            </service>
            <service>
                <role>WEBHCAT</role>
                <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
            </service>
            <service>
                <role>OOZIE</role>
                <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
            </service>
            <service>
                <role>WEBHBASE</role>
                <url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
            </service>
            <service>
                <role>HIVE</role>
                <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
            </service>
            <service>
                <role>RESOURCEMANAGER</role>
                <url>http://{{rm_host}}:{{rm_port}}/ws</url>
            </service>
        </topology>

avatar

avatar
Master Collaborator

@dvillarreal

I read your comments on

https://community.hortonworks.com/articles/38348/ranger-is-not-allowing-access-to-knox-resources-wh....

If you look at my topology, it contains the group section just like in your post. The only difference I could think of is that I use open-ldap as the directory server rather than AD. And the values for object class and attributes are different than AD.

<param>
    <name>main.ldapRealm.authorizationEnabled</name>
    <value>true</value>
</param>
<param>
    <name>main.ldapRealm.groupSearchBase</name>
    <value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.groupObjectClass</name>
    <value>posixgroup</value>
</param>
<param>
    <name>main.ldapRealm.groupIdAttribute</name>
    <value>cn</value>
</param>

avatar

Hi @Qi Wang

This should help you to learn by example when it comes to configuring your knox groups and how it relates to your ldapsearch. See Sample 4 specifically https://cwiki.apache.org/confluence/display/KNOX/Using+Apache+Knox+with+ActiveDirectory

Hope this helps.