Microservice applications prefer OpenID SSO by existing IDP, for example Keycloak. When these application access Hadoop Services with JWT, Knox should be able to verify and proxy to internal services. This article describe how to configure Knox JWT Provider and customize for your IDP requirement. There are two options. KnoxSSO direct integrate with IDP KnoxSSO, Knox Single Sign On service, integrates with IDP, for example Keycloak. After authenticated by IDP, Knox signs a JWT token for all microservices and Hadoop RESTful services. This solution is complex to configure and Knox could become the performance bottleneck. Knox JWTProvider accept jwt signed by other IDP Microservices still use current IDP for single sign on, and configure an extra Knox topology to accept other IDP signed JWT token. The second solution has less impact on existing architecture and Knox is only used when access Hadoop services. Here we will focus on this solution, and discuss how to configure Knox and what is the limitation. Add a JWTProvider Knox topology JWTProvider is explained https://knox.apache.org/books/knox-0-12-0/user-guide.html#JWT+Provider Current Ambari can't edit extra Knox topology, have to add it in command line. # ssh Knox node $ sudo su - knox # Add the new topology named jwt $ vim jwt.xml <topology> <gateway> <provider> <role>identity-assertion</role> <name>Default</name> <enabled>true</enabled> </provider> <provider> <role>authorization</role> <name>XASecurePDPKnox</name> <enabled>true</enabled> </provider> <provider> <role>federation</role> <name>JWTProvider</name> <enabled>true</enabled> <param> <!-- knox.token.audiences is optional --> <name>knox.token.audiences</name> <value>tokenbased</value> </param> </provider> </gateway> <!-- Add Hadoop Services allowed jwt access, here use Yarn UI as an example --> <service> <role>YARNUI</role> <url>http://{MASTER_NODE_1}:8088</url> <url>http://{MASTER_NODE_2}:8088</url> </service> <service> <role>RESOURCEMANAGER</role> <url>http://{MASTER_NODE_1}:8088/ws</url> <url>http://{MASTER_NODE_2}:8088/ws</url> </service> </topology> Add Knox Token Service in knoxsso toplogy for test <service> <role>KNOXTOKEN</role> <param> <name>knox.token.ttl</name> <value>600000</value> </param> <!-- knox.token.audiences is optional, must meet what's configured in JWTProvider --> <param> <name>knox.token.audiences</name> <value>tokenbased</value> </param> </service> Test Knox jwt toplogy Enable Knox demo ldap as knoxsso source accounts. Get a knoxsso jwt token. $ curl -ivku guest:guest-password https://{KNOX_NODE}:8443/gateway/knoxsso/knoxtoken/api/v1/token {"access_token":"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI4MjgxODQ3fQ.i2Y3MMGbTW9M-wHStL7TuwjmL_rYmTGRjN_7QK0KB8EfLxKJzL2zRFEU8USxyFAchDJ-3vDdLaU8UPsTCVufo9UT5p8ywSlBgulFsOzIYuq-YVIqATpJZVZIJWCnoGHjXuTZHXeRreyjAs6cFsiiqsDwL8rCxnAmtBQeoX9fsAI","token_type":"Bearer ","expires_in":1528281847804} Use this token to access Yarn UI $ curl -ivk -H "Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJndWVzdCIsImF1ZCI6InRva2VuYmFzZWQiLCJpc3MiOiJLTk9YU1NPIiwiZXhwIjoxNTI4MjgxODQ3fQ.i2Y3MMGbTW9M-wHStL7TuwjmL_rYmTGRjN_7QK0KB8EfLxKJzL2zRFEU8USxyFAchDJ-3vDdLaU8UPsTCVufo9UT5p8ywSlBgulFsOzIYuq-YVIqATpJZVZIJWCnoGHjXuTZHXeRreyjAs6cFsiiqsDwL8rCxnAmtBQeoX9fsAI" https://hdp-e.field.hortonworks.com:8443/gateway/jwt/yarn/ admin should be able to view the UI html contents, and guest would be blocked. Also can find the entries in Ranger Audit. Decode knoxsso jwt token Decode the jwt https://jwt.io/ HEADER { "alg": "RS256" } PAYLOAD { "sub": "guest", "aud": "tokenbased", "iss": "KNOXSSO", "exp": 1528281847 } Microservice Application JWT Verification Synchronise certificates between Knox and IDP IDP use private certificate to sign JWT, and Knox use public certificate to verify the JWT. The certificate pair must match for the verification. In current Knox version, public key is not supported, have to configure public certificate as following example. Knox only supports keypair JWT signature algorithms, RS256, RS384, RS512, PS256, PS384, PS512. RS256 is default and not configurable in HDP2.6. Get public certificate from IDP. $ cat knox-pub.pem -----BEGIN CERTIFICATE----- MIIENTCCAx2gAwIBAgIJAP4/owzmw1t4MA0GCSqGSIb3DQEBCwUAMIGlMQswCQYD VQQGEwJVSzEABC0GA1UECAwGTG9uZG9uMQ8wDQYDVQQHDAZMb25kb24xFDASBgNV BAoMC0hvbWUgT2ZmaWNlMUQwQgYDVQQDDDtwcmltYXJ5LXNlY3VyaXR5MC5ub25w cm9kZi51ay5zZHAuZGlnaXRhbC5ob21lb2ZmaWNlLmdvdi51azEYMBYGA1UdEQwP RE5TLjE9bG9jYWxob3N0MB4XDTE4MDUzMTE2MTA0MFoXDTI4MDUyODE2MTA0MFow gaUxCzAJBgNVBAYTAlVLMQ8wDQYDVQQIDAZMb25kb24xDzANBgNVBAcMBkxvbmRv bjEUMBIGA1UECgwLSG9tZSBPZmZpY2UxRDBCBgNVBAMMO3ByaW1hcnktc2VjdXJp dHkwLm5vbnByb2RmLnVrLnNkcC5kaWdpdGFsLmhvbWVvZmZpY2UuZ292LnVrMRgw FgYDVR0RDA9ETlMuMT1sb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDNb8qkTY0afOAUauLWbLZfF/1kys6Il/aRwKLaf0m+nTiaYowCGBKB XLbzkXXDNSCOC2b6qPRIul2yF1fd8hAmnfUiVJY2hKbbfhdvbPp1oF9cC6QL+28x ppj+utmyf2YNYpXGCMKHh7wZHNTTt28jqB/+Co8RC2xQgQ6FX6rCmUSB62/VuMHp JBCb4Sf3Y9qgiZKyqtK9B9UiNIo4bJqeyF3Ql3qCFKmXIoni4k/3wcKvw9wNiWPy MQweK0Q7G542K02Q3h7+caRNnZibFSTC/Qvs5jsRrxmkw4vH3npEJGASVr+y9JGq dCjsi+ocrEdN9SMnGlQpWWtuB8LRlsitAgMBAAGjZjBkMBQGA1UdEQQNMAuCCWxv Y2FsaG9zdDAdBgNVHQ4EFgQUPlLDl7837zNPS2Dvn2u7mtqDTZgwHwYDVR0jBBgw FoAUPlLDl7837zNPS2Dvn2u7mtqDTZgwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0B AQsFAAOCAQEAZa7ZJfc8MzwmYCCmVt9xcGQqFeAxC4saqKEFuS6PJAJqlZIK+D/A y+3AT2qJ84Rs3nWFdnIsEmGzWwLbfve/xyFFAizM8d1tYF5DxXWp+7f2c1Ssah+S t+ua80N9Q2EwdZekQrtnfba58gW5RdTORNGAVjXJjQvHSytwkn1YlRsVQjhvv4Cy R6LBb5Xdd0R9DIqu2mpp0bGvX6hlx0yPJrsiYxd1DsHl+aFdTnQ3OkZVvxe2MfWi yhTIWQoLfHrMwc2l1qjn2c3x4AIRsqLiLkMTfgrgUWC+T2IL1oO5jFBjbeV3ljOY fOfkGmG6TCsdF38qkB/fl869jUGXIBAHjQ== -----END CERTIFICATE----- In Knox JWT topology, configure the verification public certificate. <provider> <role>federation</role> <name>JWTProvider</name> <enabled>true</enabled> <param> <name>knox.token.verification.pem</name> <value> MIIENTCCAx2gAwIBAgIJAP4/owzmw1t4MA0GCSqGSIb3DQEBCwUAMIGlMQswCQYD VQQGEwJVSzEABC0GA1UECAwGTG9uZG9uMQ8wDQYDVQQHDAZMb25kb24xFDASBgNV BAoMC0hvbWUgT2ZmaWNlMUQwQgYDVQQDDDtwcmltYXJ5LXNlY3VyaXR5MC5ub25w cm9kZi51ay5zZHAuZGlnaXRhbC5ob21lb2ZmaWNlLmdvdi51azEYMBYGA1UdEQwP RE5TLjE9bG9jYWxob3N0MB4XDTE4MDUzMTE2MTA0MFoXDTI4MDUyODE2MTA0MFow gaUxCzAJBgNVBAYTAlVLMQ8wDQYDVQQIDAZMb25kb24xDzANBgNVBAcMBkxvbmRv bjEUMBIGA1UECgwLSG9tZSBPZmZpY2UxRDBCBgNVBAMMO3ByaW1hcnktc2VjdXJp dHkwLm5vbnByb2RmLnVrLnNkcC5kaWdpdGFsLmhvbWVvZmZpY2UuZ292LnVrMRgw FgYDVR0RDA9ETlMuMT1sb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDNb8qkTY0afOAUauLWbLZfF/1kys6Il/aRwKLaf0m+nTiaYowCGBKB XLbzkXXDNSCOC2b6qPRIul2yF1fd8hAmnfUiVJY2hKbbfhdvbPp1oF9cC6QL+28x ppj+utmyf2YNYpXGCMKHh7wZHNTTt28jqB/+Co8RC2xQgQ6FX6rCmUSB62/VuMHp JBCb4Sf3Y9qgiZKyqtK9B9UiNIo4bJqeyF3Ql3qCFKmXIoni4k/3wcKvw9wNiWPy MQweK0Q7G542K02Q3h7+caRNnZibFSTC/Qvs5jsRrxmkw4vH3npEJGASVr+y9JGq dCjsi+ocrEdN9SMnGlQpWWtuB8LRlsitAgMBAAGjZjBkMBQGA1UdEQQNMAuCCWxv Y2FsaG9zdDAdBgNVHQ4EFgQUPlLDl7837zNPS2Dvn2u7mtqDTZgwHwYDVR0jBBgw FoAUPlLDl7837zNPS2Dvn2u7mtqDTZgwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0B AQsFAAOCAQEAZa7ZJfc8MzwmYCCmVt9xcGQqFeAxC4saqKEFuS6PJAJqlZIK+D/A y+3AT2qJ84Rs3nWFdnIsEmGzWwLbfve/xyFFAizM8d1tYF5DxXWp+7f2c1Ssah+S t+ua80N9Q2EwdZekQrtnfba58gW5RdTORNGAVjXJjQvHSytwkn1YlRsVQjhvv4Cy R6LBb5Xdd0R9DIqu2mpp0bGvX6hlx0yPJrsiYxd1DsHl+aFdTnQ3OkZVvxe2MfWi yhTIWQoLfHrMwc2l1qjn2c3x4AIRsqLiLkMTfgrgUWC+T2IL1oO5jFBjbeV3ljOY fOfkGmG6TCsdF38qkB/fl869jUGXIBAHjQ== </value> </param> </provider> JWT Issuer By default Knox only accept JWT with "iss": "KNOXSSO". Need to verify if IDP, for example Keycloak can customize it. Alternatively, can override this in Knox configuration. Add following parameters in Ambari Knox Custom gateway-site jwt.expected.issuer=CUSTOM_SSO jwt.expected.sigalg=RS256 However this new feature is only available in HDP3.x. Before upgrade to HDP3, have to hard code jwt issuer as "KNOXSSO". User Account ID Current Knox can only parse user account from jwt parameter "sub": "guest". If IDP, for example Ping Federation, uses "client_id", it would be a problem. And the value should be consistent with account id synchronized into Ranger. Normally it is uid in OpenLDAP. JWT audiences This is optional. It must meet what's configured in knox.token.audiences Known Issues Knox json parser issue Before HDP2.6.5, Knox json parser doesn't support complex json raw format with multiple keys. Knox treats it as invalid json format, and throw NullpointException. Workaround JSON Issue For HDP2.6.3, manually replace dependant JSON jar files. SSH into every Knox node. $ cd /usr/hdp/2.6.3.0-235/knox/dep $ sudo wget http://repo1.maven.org/maven2/com/nimbusds/nimbus-jose-jwt/4.41.2/nimbus-jose-jwt-4.41.2.jar $ sudo wget http://repo1.maven.org/maven2/com/jayway/jsonpath/json-path/2.4.0/json-path-2.4.0.jar $ sudo wget http://repo1.maven.org/maven2/net/minidev/json-smart/2.3/json-smart-2.3.jar $ sudo wget http://repo1.maven.org/maven2/net/minidev/asm/1.0.2/asm-1.0.2.jar $ sudo mv json-path-0.9.1.jar json-path-0.9.1.jar.bak $ sudo mv nimbus-jose-jwt-4.11.jar nimbus-jose-jwt-4.11.jar.bak $ sudo mv json-smart-1.2.jar json-smart-1.2.jar.bak Restart Knox from Ambari ... View more

Short Description How to setup popular SQL development tools, like DbVisualizer, to access HDP Hive in secured network zone via Knox. Article Most customers setup Hadoop/Hive cluster in secured network Zone. There's no direct network connection between office and Hadoop network. So Knox is always setup to proxy Hive connections from office network. This article will setup DbVisualizer to connect to Knox Hive URL step by step. Download Hive Driver for HDP2.6.1 Download hive-jdbc.jar https://github.com/timveil/hive-jdbc-uber-jar/releases The latest for HDP2.6.1 is https://github.com/timveil/hive-jdbc-uber-jar/releases/download/v1.6-2.6.1/hive-jdbc-uber-2.6.1.0-129.jar Create a new Database Driver use the hive-jdbc.jar URL Format: jdbc:hive2://<server>:<port10000>/<database> The URL Format is for standard hive2 jdbc, will change later in actual connection for Knox. Create a new Connection with this new Driver Use Database URL, and edit the Database URL as jdbc:hive2://<KNOX_NODE_FQDN>:<port>/;ssl=true;transportMode=http;httpPath=<KNOX_HIVE_HTTPPATH> Default knox hive http path is gateway/default/hive, but please double check with your system admin. Add Knox node certificate or CA certificate into DbVisualizer JVM truststore, otherwise will get SSL Exception during DB connection. Check the Java Home of DbVisualizer. Then add Knox node certificate or CA certificate into the truststore $ sudo keytool -import -alias knox -file wb-e.crt.pem -keystore /Library/InternetPlug-Ins/JavaAppletPlugin.plugin/Contents/Home/lib/security/cacerts Enter keystore password:changeit Enjoy DbVisualizer ... View more

LDAP security options LDAP over SSL Traditionally, LDAP connection is wrapped with SSL/TLS, and via a separate service port, normally 636. The protocol is ldaps://. To be aware, it’s not just over SSL, depends on configuration, can over TLS as well. LDAP with StartTLS Alternatively, StartTLS extended operation enable encrypted connection after the initial unencrypted connection, via the same plain ldap port, normally 389. If you don’t configure properly, it could be over SSL, not always with TLS. This protocol is new, normally is used to upgrade the plain ldap port 389 to be secure. But not all customer have configured it in AD/LDAP server. Most common cases are still use ldaps for secure ldap connection. So this article is focusing on LDAPS. Components support LDAPS Most of HDP/HDF components support LDAPS. Ambari, NiFi, Ranger Usersync, Ranger Admin, Zeppelin, Knox. But Atlas still only supports LDAP. General Configure LDAPS Steps Verify LDAPS certificate To connect to ldaps service, first thing we need to get the AD/LDAPS CA certificate. And should verify it’s the correct certificate. $ openssl s_client -CAfile ${LDAPS_SERVER_CA_CERTIFICATE_FILE} -connect ${LDAPS_SERVER_FQDN}:636 -showcerts -verify 32 For example $ openssl s_client -CAfile ./AD-CA.crt -connect ad1.example.com:636 -showcerts -verify 32 Expect return "Verify return code: 0 (ok)". Otherwise need to check with the AD/LDAP team to find the correct certificate file or check the certificate file format. Test ldapsearch Then we need to use the certificate and ldap bind account to test ldapsearch, to verify we have the correct credential of the binding account. $ LDAPTLS_CACERT=${LDAPS_SERVER_CA_CERTIFICATE_FILE} -H ldaps://${LDAPS_SERVER_FQDN}:636 -D ${LDAP_BIND_USER_ACCOUNT} -b ${BASE_OU} "(&(objectclass=person)(sAMAccountName=*))" For example $ LDAPTLS_CACERT=./AD-CA.crt ldapsearch -H ldaps://ad1.example.com:636 -D "hadoopldapuser@datalake.hadoop.internal" -b "OU=Hadoop,DC=DataLake,DC=Hadoop,DC=Internal" "(&(objectclass=person)(sAMAccountName=*))" Expect return some AD/LDAP accounts. Add the certificate into Truststore $ keytool -import -file ${LDAPS_SERVER_CA_CERTIFICATE_FILE} -alias LDAPS-CA -keystore truststore.jks For example $ keytool -import -file ./AD-CA.crt -alias LDAPS-CA -keystore truststore.jks Configure components truststore and ldaps Components have slightly different configure for the ldaps. But normally need to configure the truststore and truststore password, and base OU, bind account, search filters. Detail configurations will be discussed in other articles. ... View more

When NiFi flow runs in mission critical production environment, customer would concern about how to change NiFi flow logic but don't impact business. One solution was implemented in customer production is as follow steps: 1. Add an extra non-logic processor, for example "UpdateAttribute", between ListenPort and main ProcessGroup 2. Updated NewProcessGroup, add into canvas 3. Stop "UpdateAttribute" process, make sure RunningProcessGroup consume all flowfiles in the previous queue. But client components can still send messages to Listen Port 4. After all queued flowfiles are processed by RunningProcessGroup, move the queue end connection to the NewProcessGroup 5. Start "UPdateAttribute" process and NewProcessGroup 6. At the end, remove the old RunningProcessGroup Follow these steps, the production flow change is transparent for client components, and guarantee no data lost. ... View more

Background After Kerberos HDP/HDF cluster, some native admin UI will return "401 Authentication required". For example, Ambari Infra Solr Admin UI This is because the web UI is configured for SPNEGO (Simple and Protected GSSAPI Negotiation Mechanism) to extend Kerberos to HTTP. Client must have a valid Kerberos ticket and send by browser. Mac Kerberos Client Configuration Kerberos software is installed by default in Mac OS, but need to add configure file to access your KDC server. $ sudo vi /Library/Preferences/edu.mit.Kerberos [domain_realm] .hortonworks.com = "HORTONWORKS.COM" hortonworks.com = "HORTONWORKS.COM" [libdefaults] default_realm = "HORTONWORKS.COM" dns_fallback = "yes" noaddresses = "TRUE" [realms] HORTONWORKS.COM = { admin_server = "ad.hortonworks.com" default_domain = "hortonworks.com" kdc = "ad.hortonworks.com" } Firefox Configuration type about:config search negotiate-auth Option1 Command Line command line to init a kerberos ticket $ kinit hadoopadmin@HORTONWORKS.COM hadoopadmin@HORTONWORKS.COM's password: list the ticket $ klist Option2 Ticket Viewer Go to Ticket Viewer Folder CoreServices Go to the folder: /System/Library/CoreServices/ Find Ticket Viewer Add Identity Then you can see the identity Access UI again ... View more