KNOX - Kerberos authentication in admin topology - Ranger access problem

[#1]

Hi,

We have an HDP 2.6.1.0-129 cluster on Red Hat Linux 7. The cluster is kerberized against Active Directory. We also have AD integration (SSSD) at the Linux level. Ranger is used for centralized policy management and AD is used as user/group source for Ranger.

We decided to set up SPNEGO autentication for the KNOX admin topology so that ranger can do service lookup just like for other services (e.g. HDFS).

We used the follwing configuration in /etc/knox/conf/topologies/admin.xml:

<topology>
...
      <provider>
         <role>authentication</role>
         <name>HadoopAuth</name>
         <enabled>true</enabled>
         <param>
            <name>config.prefix</name>
            <value>knox.config</value>
         </param>
         <param>
            <name>knox.config.type</name>
            <value>kerberos</value>
         </param>
         <param>
            <name>knox.config.simple.anonymous.allowed</name>
            <value>false</value>
         </param>
         <param>
            <name>knox.config.token.validity</name>
            <value>1800</value>
         </param>
         <param>
            <name>knox.config.kerberos.principal</name>
            <value>HTTP/hdphost03.mydomain.local@MYDOMAIN.LOCAL</value>
         </param>
         <param>
            <name>knox.config.kerberos.keytab</name>
            <value>/etc/security/keytabs/spnego.service.keytab</value>
         </param>
         <param>
            <name>knox.config.cookie.path</name>
            <value>gateway/admin</value>
         </param>
         <param>
            <name>knox.config.kerberos.name.rules</name>
            <value>RULE:[2:$1@$0](rangeradmin@MYDOMAIN.LOCAL)s/.*/rangerlookup/
DEFAULT</value>
         </param>
      </provider>
...
</topology>

The first problem is that Ranger uses the rangeradmin principal instead of rangerlookup for KNOX. I added a rule to knox.config.kerberos.name.rules to translate the names but I've been wondering if there's a way to force Ranger to use the lookup account. All other configured services are accessed as rangerlookup. Is there any way to change that?

The other problem is that this setup works only for some time (haven't tested precisely how long). After some time my Ranger Admin is unable to authenticate to KNOX. To make it working again, I need to restart both services.

When we obseve the problem the following entries can be found in related log files:

* KNOX: gateway-audit.log

17/10/30 16:41:40 ||03dc067a-085b-4ce1-9a03-61cc45a1cd07|audit|KNOX||||access|uri|/gateway/admin/api/v1/topologies|unavailable|Request method: GET
17/10/30 16:41:40 ||03dc067a-085b-4ce1-9a03-61cc45a1cd07|audit|KNOX||||access|uri|/gateway/admin/api/v1/topologies|success|Response status: 401 

* KNOX: nothing is written to gateway.log

* RANGER: xa_portal.log

2017-10-30 16:41:40,947 [timed-executor-pool-0] ERROR apache.ranger.services.knox.client.KnoxClient (KnoxClient.java:131) - Got invalid REST response from: https://hdphost03.mydomain.local@MYDOMAIN.LOCAL/gateway/admin/api/v1/topologies, responseStatus: 401

I can't now enable debug logging for the services, but will try when possible. I also don't have access to the KDC server now.

Has anybody tired such a setup? Any ideas how can we channel our efforts?

[#2]

Hi @Pit Err,

are you managing your admin topology by Ambari UI?

In knox service repo, we configured the service url as "https://pravin1.openstacklocal:8443/gateway/admin/api/v1/topologies" Which is an admin topology explicitly used to list the topology names in Knox. And authentication for this usually is configured for LDAP, which you can verify from admin topology like below.

=== 
<provider> 
<role>authentication</role> 
<name>ShiroProvider</name> 
<enabled>true</enabled> 
<param> 
=== 

So authentication credentials should be set based on admin topology "authentication" module in admin.xml (or from ambari Advanced admin-topology).

Also : URL doesn't seem correct to me.

https://hdphost03.mydomain.local@MYDOMAIN.LOCAL/gateway/admin/api/v1/topologies, responseStatus: 401

Please check the knox.url * in the service repo . Also, are you using Ranger Acl for KNOX?
One more thing, Ranger Test Connection is an additional feature to test. To confirm ranger acl are working properly for knox you need to check access logs of Ranger.


If you want to use "rangerlookup" for Ranger KNOX plugin. You need to specify accordingly.

Lookup is handled by the user which is configured in the Service Repository in Ranger UI, Also check that user has the policy to do the hive query. It has to be hadoop user for the user to get authorized. In secure cluster it has to be principal with password from kdc i.e. hive@EXAMPLE.COM

or in your case rangerlookup Prepare Ranger Lookup

[#3]

Hi,

With regard to: "are you managing your admin topology by Ambari UI?".

Yes, we manage the topology with Ambari UI, but this has little to do with the authentication method used in it. As you mentioned, admin topology authentication is often LDAP. For certain reasons in our environment we decided to change that to SPNEGO and use Kerberos. The configuration used is as listed in my previous post.

With regard to: "URL doesn't seem correct to me."

You're right. I made a mistake while masking some customer specific names. The correct entry (with masked names) is like below:

2017-10-3016:41:40,947[timed-executor-pool-0] ERROR apache.ranger.services.knox.client.KnoxClient(KnoxClient.java:131)-Got invalid REST response from: https://hdphost03.mydomain.local:8443/gateway/admin/api/v1/topologies, responseStatus: 401

Regarding: "Also, are you using Ranger Acl for KNOX?"

Yes we do and yes we have tested it multiple times with the "test connection" button. As written earlier, the problem is that this setup works only for some time (haven't tested precisely how long). After some time my Ranger Admin is unable to authenticate to KNOX. To make it working again, I need to restart both services. Corresponding log entries are also visible above.

Regarding: "If you want to use "rangerlookup" for Ranger KNOX plugin. You need to specify accordingly. Lookup is handled by the user which is configured in the Service Repository in Ranger UI"

That is not fully true. If an environment is Kerberized, Ranger uses the principal configured as "ranger.lookup.kerberos.principal" (check below).

root@hdphost02:/etc/ranger/admin/conf$ grep -A1 ranger.lookup.kerberos ranger-admin-site.xml
      <name>ranger.lookup.kerberos.keytab</name>
      <value>/etc/security/keytabs/rangerlookup.service.keytab</value>
--
      <name>ranger.lookup.kerberos.principal</name>
      <value>rangerlookup/_HOST@MYDOMAIN.LOCAL</value>

This works correctly for all services but Knox. The question is how to force Ranger to use it also for KNOX.

Regards,

Pit