Support Questions

qiwang · ‎01-18-2017

HDP 2.5 secured cluster with Knox installed as gateway. Ranger policy is create for default topology and WEBHDFS service. However when the policy only contains only hr group which hr1 user is part of, I got error from gateway request

curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /gateway/default/webhdfs/v1/hr/exempt. Reason:
<pre>    Forbidden</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>


</body>
</html>

When user hr1 is included in the policy, it works fine

curl -ik -u hr1 https://<knox-gateway>:8443/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS


{"FileStatuses":{"FileStatus":[{"accessTime":1483632050751,"blockSize":134217728,"childrenNum":0,"fileId":152421,"group":"hr","length":23,"modificationTime":1483632051087,"owner":"hdfs","pathSuffix":"testfile","permission":"644","replication":3,"storagePolicy":0,"type":"FILE"}]}}

This happens with other knox services as well like hive through knox. The group policy does work fine on other Ranger policies not for knox, like HDFS and Hive. Only the policies for Knox seems to have this particular problem.

And in HDFS custom core-site, I have

hadoop.proxyuser.knox.hosts=*
hadoop.proxyuser.knox.groups=*

dvillarreal · ‎01-19-2017

Hi @Qi Wang

This should help you to learn by example when it comes to configuring your knox groups and how it relates to your ldapsearch. See Sample 4 specifically https://cwiki.apache.org/confluence/display/KNOX/Using+Apache+Knox+with+ActiveDirectory

Hope this helps.

View solution in original post

lmccay · ‎01-18-2017

More than likely, you do not have group lookup configured in Knox. If you check the {GATEWAY_HOME}/logs/gateway-audit.log you will likely notice an empty array "[]" for groups with the authentication entries.

The groups need to be looked up by the Knox code and made available to the Ranger Knox plugin. The plugin doesn't do its own group lookup.

HTH

qiwang · ‎01-18-2017

@lmccay

As you mentioned, the log does indicate the group search is no returning the right group

17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|
17/01/18 15:31:26 ||5725e8ba-938d-40a7-86b9-64642ad8903f|audit|WEBHDFS|hr1|||authentication|uri|/gateway/default/webhdfs/v1/hr/exempt?op=LISTSTATUS|success|Groups: []

How do I config knox to do group lookup? Don't see anything in my topology related to group lookup

<topology>
            <gateway>
                <provider>
                    <role>authentication</role>
                    <name>ShiroProvider</name>
                    <enabled>true</enabled>
                    <param>
                        <name>sessionTimeout</name>
                        <value>30</value>
                    </param>
                    <param>
                        <name>main.ldapRealm</name>
                        <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value> 
                    </param>


<!-- changes for AD/user sync -->

<param>
    <name>main.ldapContextFactory</name>
    <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>

<!-- main.ldapRealm.contextFactory needs to be placed before other main.ldapRealm.contextFactory* entries  -->
<param>
    <name>main.ldapRealm.contextFactory</name>
    <value>$ldapContextFactory</value>
</param>

<!-- AD url -->
<param>
    <name>main.ldapRealm.contextFactory.url</name>
    <value>ldap://qwang-kdc-ldap.field.hortonworks.com:389</value> 
</param>

<!-- system user -->
<param>
    <name>main.ldapRealm.contextFactory.systemUsername</name>
    <value>cn=admin,dc=field,dc=hortonworks,dc=com</value>
</param>

<!-- pass in the password using the alias created earlier -->
<param>
    <name>main.ldapRealm.contextFactory.systemPassword</name>
    <value>password</value>
</param>

                    <param>
                        <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                        <value>simple</value>
                    </param>
                    <param>
                        <name>urls./**</name>
                        <value>authcBasic</value> 
                    </param>


<!--  AD groups of users to allow -->
<param>
    <name>main.ldapRealm.searchBase</name>
    <value>ou=Users,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.userObjectClass</name>
    <value>person</value>
</param>
<param>
    <name>main.ldapRealm.userSearchAttributeName</name>
    <value>uid</value>
</param>


<!-- changes needed for group sync-->
<param>
    <name>main.ldapRealm.authorizationEnabled</name>
    <value>true</value>
</param>
<param>
    <name>main.ldapRealm.groupSearchBase</name>
    <value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.groupObjectClass</name>
    <value>posixgroup</value>
</param>
<param>
    <name>main.ldapRealm.groupIdAttribute</name>
    <value>cn</value>
</param>
                </provider>
                <provider>
                    <role>identity-assertion</role>
                    <name>Default</name>
                    <enabled>true</enabled>
                </provider>
                <provider>
                    <role>authorization</role>
                    <name>XASecurePDPKnox</name>
                    <enabled>true</enabled>
                </provider>
            </gateway>
            
            <service>
                <role>NAMENODE</role>
                <url>hdfs://{{namenode_host}}:{{namenode_rpc_port}}</url>
            </service>
            <service>
                <role>JOBTRACKER</role>
                <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
            </service>
            <service>
                <role>WEBHDFS</role>
                <url>http://{{namenode_host}}:{{namenode_http_port}}/webhdfs</url>
            </service>
            <service>
                <role>WEBHCAT</role>
                <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
            </service>
            <service>
                <role>OOZIE</role>
                <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
            </service>
            <service>
                <role>WEBHBASE</role>
                <url>http://{{hbase_master_host}}:{{hbase_master_port}}</url>
            </service>
            <service>
                <role>HIVE</role>
                <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
            </service>
            <service>
                <role>RESOURCEMANAGER</role>
                <url>http://{{rm_host}}:{{rm_port}}/ws</url>
            </service>
        </topology>

dvillarreal · ‎01-18-2017

Hi @Qi Wang,

This may also help where I have answered a similar question.

https://community.hortonworks.com/questions/74501/how-knox-pass-the-user-information-to-ranger.html

qiwang · ‎01-18-2017

@dvillarreal

I read your comments on

https://community.hortonworks.com/articles/38348/ranger-is-not-allowing-access-to-knox-resources-wh....

If you look at my topology, it contains the group section just like in your post. The only difference I could think of is that I use open-ldap as the directory server rather than AD. And the values for object class and attributes are different than AD.

<param>
    <name>main.ldapRealm.authorizationEnabled</name>
    <value>true</value>
</param>
<param>
    <name>main.ldapRealm.groupSearchBase</name>
    <value>ou=Groups,dc=field,dc=hortonworks,dc=com</value>
</param>
<param>
    <name>main.ldapRealm.groupObjectClass</name>
    <value>posixgroup</value>
</param>
<param>
    <name>main.ldapRealm.groupIdAttribute</name>
    <value>cn</value>
</param>

dvillarreal · ‎01-19-2017

Hi @Qi Wang

This should help you to learn by example when it comes to configuring your knox groups and how it relates to your ldapsearch. See Sample 4 specifically https://cwiki.apache.org/confluence/display/KNOX/Using+Apache+Knox+with+ActiveDirectory

Hope this helps.

Cloudera Community

Support Questions

HDP 2.5 Ranger Knox policy only works with user