KnoxSSO Integration with Identity Provider (Ping Federate)

[#1]

Introduction

This document describes the details of Knox SSO integration with a SAML identity provider (For the purpose of this article, the IDP is Ping Federate). Here, Ping Federate acts as the identity provider and it has the ability to connect to any authentication source such as AD, LDAP, etc.

The following diagram depicts the process at a high level: 

The SAML authentication process has three components 

• Identity Provider: Identity provider has the flexibility to authenticate users against an authentication source, and sends a SAML response XML back to the service provider
• Service Provider: Service provider is an application that requires users to authenticate
• Authentication source:  A directory service such as AD, LDAP. The service provider uses the authentication token and does not need to know the authentication source.

As part of this integration, both IDP (Ping Fed) and SP (Knox server) exchange information for the handshake to work successfully.

The following information is shared to IDP by SP:

• Service Provider EntityID= https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Clien...
• Assertion Consumer service URL (IDP sends SAML response to this endpoint) = https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Clien...
• Knox hostnames and IP addresses so that ports can be opened on the firewalls etc 

The following information is shared to SP by IDP:

• IDP metadata URL

Knox configuration

1. Configure knoxsso.xml to use pac4j provider which has the identity provider metadata that allows Knox to communicate with IDP. A sample knoxsso XML is available in the Appendix section below.
2. Configure the Service XML that uses the SSOCookieProvider mechanism to route the initial service request to KnoxSSO.
3. Request the IDP team for the metadata XML file. You may receive the metadata URL from the IDP team. This URL can be added in the Knox SSO XML under the property:

   saml.identityProviderMetadataPath​

4. Conversely, the IDP Metadata URL can be entered on the browser. Copy the XML content and paste it in a file such as /etc/knox/conf/idp-metadata.xml. This file path can be used as a value for the property saml.identityProviderMetadataPath.
5. Ensure that the file is owned by Knox:

   $ chown knox:knox /etc/knox/conf/idp-metadata.xml​

6. The saml.serviceProviderMetadataPath property should be set to a file path on Knox server such as /etc/knox/conf/sp-metadata.xml. The property should be in knoxsso.xml file however, the file does not exist.
7. Ensure that the service provider EntityID is shared to IDP team and it matches what we have in knoxsso.xml configuration.
8. The Service Provider EntityID look like the following:

   https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client​

    In our scenario, the above URL was shared with the IDP team for them to the update SP metadata. However, we ran into an issue where the Entity IDs did not match between SP & IDP. The SAML trace shows that the “amp;” was removed.
   The solution for this is to remove “amp;” from the SP Entity ID on IDP:
   SP Entity ID on knox:

   https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client​

   
   SP Entity ID shared to Ping Federate: 

   https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&client_name=SAML2Client​

9. Create a Knox topology XML that uses the SSOCookieProvider and lists the services that should be placed behind the Knox SSO. The example file is present in the Appendix section.
10. Ensure that the knoxsso topology content has been added in Ambari Knox configuration in the Advanced-Knoxsso-topology section and restart the Knox service.
11. Ensure that both topology files (knoxsso, service topology xml) are owned by the Knox user: 

    $ chown knox:knox /etc/knox/conf/topologies/knoxsso.xml
    
    $ chown knox:knox /etc/knox/conf/topologies/clustername-sso.xml (this file is in Appendix)​

Issues Encountered:

ISSUE 1

The attempt to access a Knox URL failed with below error while Knox was trying to communicate to Ping Federate server:

javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException: java.lang.UnsupportedOperationException: trusted certificate entries are not password-protected

This error indicates that the keystore has a certificate that is of type “trustedCertEntry”.  Pac4j expects all certificates in Knox keystore to be of type “PrivateKeyEntry”.

The certificate was converted from "trustedCertEntry" to "PrivateKeyEntry".  To resolve this issue, do the following to create a PrivateKeyEntry certificate: 

1. Obtain the public key and private key for the certificate.
2. Create a PKCS12 formatted keystore from the above keys:

   $ openssl pkcs12 -export -in cert -inkey key -out pkcert.p12 -name pkcert.alias​

3. Import the cert from PKCS12 formatted cert into the Knox java keystone:

   $ keytool -importkeystore -deststorepass knoxgwpasswd -destkeypass knoxgwkeypass -destkeystore gateway.jks -srckeystore pkcert.p12 -srcstoretype PKCS12 -srcstorepass somepasswd -alias pkcert.alias​

4. Delete any certificates of kind "trustedCertEntry" from knox gateway.jks:

   $ keytool -delete -noprompt -alias xxxxxx  -keystore gateway.jks -storepass xxxxxx​

5. Once this is done, restart Knox and test again by entering a Knox URL for a Hadoop service (Ranger, Atlas, Yarn UI, Spark UI, Zeppelin UI, etc. )

ISSUE 2

Another error encountered during our test is that upon entering the URL for service UI (yarn UI), the browser redirects to the IDP login page, and upon entering credentials, it takes the user back to the IDP login page.

This issue occurred because Knox certificate was regenerated or recreated and the Knox public key was not replaced in the configuration of services like Yarn, MapReduce, Ranger, Spark, Atlas, etc.

To resolve this issue, ensure that whenever Knox certificate or Keystore is modified, the public key should be extracted and added into Yarn and other service configurations. Once all the above configuration is in place, the Knox SSO service URL can be entered on the browser:

https://knoxhost.domain:8443/gateway/topology/yarn/

The topology in above URL is not the KNOXSSO topology. It should be another topology that has SSOCookieProvider and has service URL mappings.

Once you enter the local AD credentials, IDP (Ping Fed) validates the user credentials and sends the SAML response back to Knox. Knox SSO redirects back to Yarn UI. 

Appendix

The following are the Knox gateway logs during successful round trips form Knox to Ping federate:

Knoxsso.xml

<topology>
        <gateway>
                <provider>
                        <role>federation</role>
                        <name>pac4j</name>
                        <enabled>true</enabled>
                        <param>
                                <name>pac4j.callbackUrl</name>                                <value>https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso</value>
                        </param>
                        <param>
                                <name>clientName</name>
                                <value>SAML2Client</value>
                        </param>
                        <param>
                                <name>saml.identityProviderMetadataPath</name>
                                <value>/etc/knox/conf/idp-metadata.xml</value>
                        </param>
                        <param>
                                <name>saml.serviceProviderMetadataPath</name>
                                <value>/etc/knox/conf/sp-metadata.xml</value>
                        </param>
                        <param>
                                <name>saml.serviceProviderEntityId</name>
<value>https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true&amp;client_name=SAML2Client</value>
                </param>
                </provider>
        </gateway>
        <service>
                <role>KNOXSSO</role>
                <param>
                        <name>knoxsso.cookie.secure.only</name>
                        <value>true</value>
                </param>
                <param>
                        <name>knoxsso.token.ttl</name>
                        <value>3600000</value>
                </param>
                <param>
                        <name>knoxsso.redirect.whitelist.regex</name>
                        <value>.*$</value>
                </param>
        </service>
</topology>

Knox Topology XML to expose the services(clustername-sso.xml): 

<topology>
    <generated>true</generated>
<gateway>
<provider>
    <role>webappsec</role>
    <name>WebAppSec</name>
    <enabled>true</enabled>
    <param>
      <name>cors.enabled</name>
      <value>true</value>
    </param>
    <param><name>xframe.options.enabled</name><value>true</value></param>
  </provider>
  <provider>
    <role>federation</role>
    <name>SSOCookieProvider</name>
    <enabled>true</enabled>
    <param>
      <name>sso.authentication.provider.url</name>
      <value>https://knoxhost.domain:8443/gateway/knoxsso/api/v1/websso</value>
    </param>
  </provider>
  <provider>
    <role>identity-assertion</role>
    <name>Default</name>
    <enabled>true</enabled>
  </provider>
</gateway>
<service>
<role>ATLAS-API</role>
<url>https://knoxhost.domain:21443</url>
<url>https://knoxhost.domain:21443</url>
</service>
<service>
<role>ATLAS</role>
<url>https://knoxhost.domain:21443</url>
<url>https://knoxhost.domain:21443</url>
</service>
<service>
<role>RANGER</role>
<url>https://knoxhost.domain:6182</url>
</service>
<service>
<role>RANGERUI</role>
<url>https://knoxhost.domain:6182</url>
</service>
<service>
  <role>YARNUI</role>
  <url>https://knoxhost.domain:8090</url>
  <url>https://knoxhost.domain:8090</url>
</service>
</topology>