Created 03-22-2019 08:16 PM
We use the following topologies right now in HDP 3, shown below. I understand there is authentication and authorization. Prior to using PAM, we used LDAP configurations. Then we ran into...
PAM works it seems, but even if I put my own username in the top level Knox policy to deny access, nothing is denied when I access Knox SSO webui's. I don't understand what part of the authorization process is not communicating. In our view, when you pass the PAM authentication stage, you just get access, when what we REALLY want to happen, is Ranger then says yes or no to you getting authorized.
PAM module:
#%PAM-1.0 auth [user_unknown=ignore success=ok ignore=ignore default=bad] pam_securetty.so #auth substack system-auth ############################################### # Imported from: /etc/pam.d/password-auth ############################################### auth required pam_env.so auth sufficient pam_unix.so nullok try_first_pass auth sufficient pam_ldap.so minimum_uid=1000 use_first_pass auth requisite pam_succeed_if.so uid >= 1000 quiet #auth sufficient pam_sss.so use_first_pass auth required pam_deny.so account required pam_unix.so broken_shadow account sufficient pam_ldap.so minimum_uid=1000 account sufficient pam_localuser.so account sufficient pam_succeed_if.so uid < 1000 quiet account [default=bad success=ok user_unknown=ignore] pam_sss.so account required pam_permit.so password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type= password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok password sufficient pam_ldap.so minimum_uid=1000 try_first_pass #password sufficient pam_sss.so use_authtok password required pam_deny.so session optional pam_keyinit.so revoke session required pam_limits.so session optional pam_mkhomedir.so umask=0077 -session optional pam_systemd.so session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_ldap.so minimum_uid=1000 #session optional pam_sss.so
knoxsso topology
<topology> <gateway> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>webappsec</role> <name>WebAppSec</name> <enabled>true</enabled> <param> <name>xframe.options.enabled</name> <value>true</value> </param> </provider> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>redirectToUrl</name> <value>/gateway/knoxsso/knoxauth/login.html</value> </param> <param> <name>restrictedCookies</name> <value>rememberme,WWW-Authenticate</value> </param> <param> <name>main.pamRealm</name> <value>org.apache.knox.gateway.shirorealm.KnoxPamRealm</value> </param> <param> <name>main.pamRealm.service</name> <value>knoxsso</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider> <provider> <role>authorization</role> <name>AclsAuthz</name> <enabled>true</enabled> </provider> </gateway> <application> <name>knoxauth</name> </application> <service> <role>KNOXSSO</role> <param> <name>knoxsso.cookie.secure.only</name> <value>true</value> </param> <param> <name>knoxsso.token.ttl</name> <value>900000</value> </param> <param> <name>knoxsso.redirect.whitelist.regex</name> <value>^https?:\/\/(.*\.DOMAIN\.COM|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$</value> </param> </service> </topology>
default topology:
<topology> <gateway> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>sessionTimeout</name> <value>30</value> </param> <param> <name>main.pamRealm</name> <value>org.apache.hadoop.gateway.shirorealm.KnoxPamRealm</value> </param> <param> <name>main.pamRealm.service</name> <value>knoxsso</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> </gateway> <service> <role>AVATICA</role> <url>http://HOST.DOMAIN.COM:8765</url> </service> <service> <role>DRUID-COORDINATOR-UI</role> {{druid_coordinator_urls}} </service> <service> <role>DRUID-COORDINATOR</role> {{druid_coordinator_urls}} </service> <service> <role>DRUID-OVERLORD-UI</role> {{druid_overlord_urls}} </service> <service> <role>DRUID-OVERLORD</role> {{druid_overlord_urls}} </service> <service> <role>DRUID-ROUTER</role> {{druid_router_urls}} </service> <service> <role>DRUID-BROKER</role> {{druid_broker_urls}} </service> <service> <role>HBASEUI</role> <url>http://HOST.DOMAIN.COM:16010</url> </service> <service> <role>HDFSUI</role> <version>2.7.0</version> <url>http://HOST.DOMAIN.COM:50070/</url> </service> <service> <role>HIVE</role> <url>http://{{hive_server_host}}:{{hive_http_port}}/{{hive_http_path}}</url> </service> <service> <role>JOBTRACKER</role> <url>rpc://{{rm_host}}:{{jt_rpc_port}}</url> </service> <service> <role>JOBHISTORYUI</role> <url>http://HOST.DOMAIN.COM:19888</url> </service> <service> <role>NAMENODE</role> <url>{{namenode_address}}</url> </service> <service> <role>OOZIE</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie</url> </service> <service> <role>OOZIEUI</role> <url>http://{{oozie_server_host}}:{{oozie_server_port}}/oozie/</url> </service> <service> <role>RANGERUI</role> <url>http://HOST.DOMAIN.COM:6080</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://{{rm_host}}:{{rm_port}}/ws</url> </service> <service> <role>SPARKHISTORYUI</role> <url>http://HOST.DOMAIN.COM:18081</url> </service> <service> <role>WEBHDFS</role> {{webhdfs_service_urls}} </service> <service> <role>WEBHCAT</role> <url>http://{{webhcat_server_host}}:{{templeton_port}}/templeton</url> </service> <service> <role>WEBHBASE</role> <url>http://{{hbase_master_host}}:60080</url> </service> <service> <role>YARNUI</role> <url>http://HOST.DOMAIN.COM:8088</url> </service> <service> <role>YARNUIV2</role> <url>http://HOST.DOMAIN.COM:8088</url> </service> <service> <role>ZEPPELINUI</role> {{zeppelin_ui_urls}} </service> <service> <role>ZEPPELINWS</role> {{zeppelin_ws_urls}} </service> </topology>
This is all I noticed in gateway.log when testing these scenarios:
[mtdeguzis@HOST: knox]$ tailf gateway.log | grep -iE 'mtdeguzis|yarn' 2019-03-22 15:49:07,094 INFO knox.gateway (KnoxPamRealm.java:doGetAuthorizationInfo(126)) - Computed roles/groups: [udaoptst3, udaops, HOST-login, mtdeguzis, cdisadmin] for principal: mtdeguzis 2019-03-22 15:49:07,405 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(240)) - About to redirect to original URL: https://HOST.domain.com:8443/gateway/knoxsso-webuis/yarnuiv2 2019-03-22 15:49:07,972 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: HTTP_ONLY, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 2019-03-22 15:49:07,973 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value HTTP_ONLY, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException 2019-03-22 15:49:08,013 ERROR knox.gateway (UrlRewriteProcessor.java:rewrite(166)) - Failed to rewrite URL: kerberos, direction: OUT via rule: YARNUIV2/yarnuiv2/outbound/timeline, status: FAILURE 2019-03-22 15:49:08,013 ERROR knox.gateway (JsonFilterReader.java:filterStreamValue(547)) - Failed to filter value kerberos, rule YARNUIV2/yarnuiv2/outbound/timeline: java.lang.NullPointerException
Created 08-09-2019 09:43 AM
Hello,
I know I'm late to the party, but I found your post while searching for a solution for my predicament.
I have a kerberised HDP3.1 cluster with Knox SSO enabled via SAML with a LemonLDAP identity provider and had the exact same error present in the Knox gateway log when trying to access YarnUIv2.
My solution consisted of 2 steps:
1. Apply the workaround for BUG-110192 described here as while debugging with the Chrome browser console (Ctrl+Shift+i ) I noticed Yarn timeline was giving out 401 HTTP error code.
2. In case you have Yarn RM HA enabled, please make sure the ACTIVE RM is the SAME as the host configured in your YARN/YARNUIV2 rules from your Knox advanced topology as currently KNOX DOES NOT SUPPORT HA for YARN and YARN UI will not work.
Hope this helps!
Created 08-09-2019 01:52 PM
The above question and the entire response thread below were originally posted in the Community Help track. On Fri Aug 9 13:51 UTC 2019, a member of the HCC moderation staff moved it to the Security track. The Community Help Track is intended for questions about using the HCC site itself.