Using Knox SSO with Shiro PAM realm, Ranger not working?

[#1]

We use the following topologies right now in HDP 3, shown below. I understand there is authentication and authorization. Prior to using PAM, we used LDAP configurations. Then we ran into...

• KnoxLdapContextFactory.java (HW had us try identity assertion, null pointer issue hit)
• KnoxLdapRealm.java (ldap - too many entries, fails, we only get the 'cdisadmin' group as it's in the first page or results, and the code for this does NOT properly page)
• KnoxPamRealm.java (works, but requires reworking PAM/sssd,).

PAM works it seems, but even if I put my own username in the top level Knox policy to deny access, nothing is denied when I access Knox SSO webui's. I don't understand what part of the authorization process is not communicating. In our view, when you pass the PAM authentication stage, you just get access, when what we REALLY want to happen, is Ranger then says yes or no to you getting authorized.

PAM module:

#%PAM-1.0
auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so
#auth       substack     system-auth
###############################################
# Imported from: /etc/pam.d/password-auth
###############################################
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        sufficient    pam_ldap.so minimum_uid=1000 use_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet
#auth        sufficient    pam_sss.so use_first_pass
auth        required      pam_deny.so

account     required      pam_unix.so broken_shadow
account     sufficient    pam_ldap.so minimum_uid=1000
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    sufficient    pam_ldap.so minimum_uid=1000 try_first_pass
#password    sufficient    pam_sss.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     optional      pam_mkhomedir.so umask=0077
-session    optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_ldap.so minimum_uid=1000
#session     optional      pam_sss.so

knoxsso topology

<topology>
  <gateway>

    <provider>
      <role>identity-assertion</role>
      <name>Default</name>
      <enabled>true</enabled>
    </provider>

    <provider>
      <role>webappsec</role>
      <name>WebAppSec</name>
      <enabled>true</enabled>
      <param>
        <name>xframe.options.enabled</name>
        <value>true</value>
      </param>
    </provider>
 
   <provider>
      <role>authentication</role>
      <name>ShiroProvider</name>
      <enabled>true</enabled>
      <param>
        <name>sessionTimeout</name>
        <value>30</value>
      </param>
      <param>
        <name>redirectToUrl</name>
        <value>/gateway/knoxsso/knoxauth/login.html</value>
      </param>
      <param>      
        <name>restrictedCookies</name>
        <value>rememberme,WWW-Authenticate</value>
      </param>
      <param>
        <name>main.pamRealm</name>
        <value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value>
      </param>
      <param>
        <name>main.pamRealm.service</name>
        <value>knoxsso</value> 
      </param>
      <param>
        <name>urls./**</name>
        <value>authcBasic</value>
      </param>
    </provider>

    <provider> 
      <role>authorization</role> 
      <name>AclsAuthz</name> 
      <enabled>true</enabled> 
    </provider> 

  </gateway>
  <application>
    <name>knoxauth</name>
  </application>

  <service>
    <role>KNOXSSO</role>
    <param>
      <name>knoxsso.cookie.secure.only</name>
      <value>true</value>
    </param>
    <param>
      <name>knoxsso.token.ttl</name>
      <value>900000</value>
    </param>
    <param>
      <name>knoxsso.redirect.whitelist.regex</name>
      <value>^https?:\/\/(.*\.DOMAIN\.COM|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value>
    </param>
  </service>

</topology>

default topology:

<topology>
  <gateway>
    <provider>
      <role>identity-assertion</role>
      <name>Default</name>
      <enabled>true</enabled>
    </provider>
    <provider>
      <role>authentication</role>
      <name>ShiroProvider</name>
      <enabled>true</enabled>
      <param>
        <name>sessionTimeout</name>
        <value>30</value>
      </param>
      <param>
        <name>main.pamRealm</name>
        <value>org.apache.hadoop.gateway.shirorealm.KnoxPamRealm</value>
      </param>
      <param>
        <name>main.pamRealm.service</name>
        <value>knoxsso</value> </param>
      <param>
        <name>urls./**</name>
        <value>authcBasic</value>
      </param>
     </provider>
     <provider>
       <role>authorization</role>
       <name>XASecurePDPKnox</name>
       <enabled>true</enabled>
     </provider>
   </gateway>
   <service>
     <role>AVATICA</role>
     <url>http://HOST.DOMAIN.COM:8765</url>
   </service>
   <service>
     <role>DRUID-COORDINATOR-UI</role>
     {{druid_coordinator_urls}}
   </service>
   <service>
     <role>DRUID-COORDINATOR</role>
     {{druid_coordinator_urls}}
   </service>
   <service>
     <role>DRUID-OVERLORD-UI</role>
     {{druid_overlord_urls}}
   </service>
   <service>
     <role>DRUID-OVERLORD</role>
     {{druid_overlord_urls}}
   </service>
   <service>
     <role>DRUID-ROUTER</role>
     {{druid_router_urls}}
   </service>
   <service>
     <role>DRUID-BROKER</role>
     {{druid_broker_urls}}
   </service>
   <service>
     <role>HBASEUI</role>
     <url>http://HOST.DOMAIN.COM:16010</url>
   </service>
   <service>
     <role>HDFSUI</role>
     <version>2.7.0</version>
     <url>http://HOST.DOMAIN.COM:50070/</url>
   </service>
   <service>
     <role>HIVE</role>
     <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url>
   </service>
   <service>
     <role>JOBTRACKER</role>
     <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url>
   </service>
   <service>
     <role>JOBHISTORYUI</role>
     <url>http://HOST.DOMAIN.COM:19888</url>
   </service>
   <service>
     <role>NAMENODE</role>
     <url>{{namenode_address}}</url>
   </service>
   <service>
     <role>OOZIE</role>
     <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url>
   </service>
   <service>
     <role>OOZIEUI</role>
     <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie/</url>
   </service>
   <service>
     <role>RANGERUI</role>
     <url>http://HOST.DOMAIN.COM:6080</url>
   </service>
   <service>
     <role>RESOURCEMANAGER</role>
     <url>http://{{rm_host}}:{{rm_port}}/ws</url>
   </service>
   <service>
     <role>SPARKHISTORYUI</role>
     <url>http://HOST.DOMAIN.COM:18081</url>
   </service>
   <service>
     <role>WEBHDFS</role>
     {{webhdfs_service_urls}}
   </service>
   <service>
     <role>WEBHCAT</role>
     <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url>
   </service>
   <service>
     <role>WEBHBASE</role>
     <url>http://{{hbase_master_host}}:60080</url>
   </service>
   <service>
     <role>YARNUI</role>
     <url>http://HOST.DOMAIN.COM:8088</url>
   </service>
   <service>
     <role>YARNUIV2</role>
     <url>http://HOST.DOMAIN.COM:8088</url>
   </service>
   <service>
     <role>ZEPPELINUI</role>
     {{zeppelin_ui_urls}}
   </service>
   <service>
     <role>ZEPPELINWS</role>
     {{zeppelin_ws_urls}}
   </service>
</topology>

This is all I noticed in gateway.log when testing these scenarios:

[mtdeguzis@HOST: knox]$ tailf gateway.log | grep -iE 'mtdeguzis|yarn' 
2019-03-22 15:49:07,094 INFO knox.gateway (KnoxPamRealm.java:doGetAuthorizationInfo(126)) - Computed roles/groups: [udaoptst3, udaops, HOST-login, mtdeguzis, cdisadmin] for principal: mtdeguzis 
2019-03-22 15:49:07,405 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(240)) - About to redirect to original URL: https://HOST.domain.com:8443/gateway/knoxsso-webuis/yarnuiv2 
2019-03-22 15:49:07,972 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: HTTP_ONLY, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 
2019-03-22 15:49:07,973 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value HTTP_ONLY, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException 
2019-03-22 15:49:08,013 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: kerberos, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 
2019-03-22 15:49:08,013 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value kerberos, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException

[#2]

Hello,

I know I'm late to the party, but I found your post while searching for a solution for my predicament.

I have a kerberised HDP3.1 cluster with Knox SSO enabled via SAML with a LemonLDAP identity provider and had the exact same error present in the Knox gateway log when trying to access YarnUIv2.

My solution consisted of 2 steps:

1. Apply the workaround for BUG-110192 described here as while debugging with the Chrome browser console (Ctrl+Shift+i ) I noticed Yarn timeline was giving out 401 HTTP error code.

2. In case you have Yarn RM HA enabled, please make sure the ACTIVE RM is the SAME as the host configured in your YARN/YARNUIV2 rules from your Knox advanced topology as currently KNOX DOES NOT SUPPORT HA for YARN and YARN UI will not work.

Hope this helps!

[#3]

The above question and the entire response thread below were originally posted in the Community Help track. On Fri Aug 9 13:51 UTC 2019, a member of the HCC moderation staff moved it to the Security track. The Community Help Track is intended for questions about using the HCC site itself.