Demystify Knox, LDAP, SSL, CA Cert integration

[#1]

This article was inspired by How to configure knox with existing ssl certificate and a client engagement to give more clarity into the actual commands that you need run to setup Apache Knox for SSL and LDAP with existing certificates.

Pre-requisites

You need to get procure the following items from your security department before you begin:

1. Your LDAP or AD Digital Certificate: <ldap>.crt
2. The company's Digital CA Cert: <company_ca>.crt
3. The certificate/ key pair for the gateway node: <gateway_node>.crt and <gateway_node>.pem
4. The passphrase for the above gateway node key.
5. Your Knox Master Secret

--------------------------------------------------------------------

What if?

1. You don't know the Knox Master Secret? Then you can change it as follows

   cd $gateway
   
   bin/knoxcli.cmd create-master --force
   
   -- Delete Keystores and restart Knox
   > rm data/security/keystores/gateway.jks
   
   > rm data/security/keystores__gateway-credentials.jceks
   

2. You don't have a signed cert from a trusted CA Authority for your Gateway Node?

   Follow the steps in Hortonworks Doc here to request one from your signing authority. They are also steps available here in the Apache Doc.

3. You Don't have time to get a trusted cert and you just want a self signed cert for evaluation? This would be the subject of another article.

   In the meantime please check out these great blogs on steps to your own self signed cert or become your own CA for evaluation purposes.

   SSL Between Knox and WebHDFS

   Deploying HTTPS in HDFS

   OR Follow the steps in the doc Self signed certificate specific hostname evaluations

-----------------------------------------------------------------------

LDAP Certificate with Apache Knox Steps

1. You would need to import your LDAP Certificate into the Java Key Store

   #First do a Key List to see if your company's signing authority is already trusted in Java Trust Store
   keytool -list -keystore ${JAVA_HOME}/jre/lib/security/cacerts | grep <Replace with Your Company's Cert Authority>
   
   Enter keystore password:#the default is 'changeit' unless someone else changed it :-)
   >changeit
   
   #If it is not there, or if in doubt, import it.
   keytool -importcert -trustcacerts -file <ldap>.crt -storepass changeit -noprompt -alias MyLdapCert -keystore ${JAVA_HOME}/jre/lib/security/cacerts

2. Follow the Hortonworks Documentation and/or Apache Knox documentation to configure Knox for LDAP.

This article is only meant to give insight into the elusive commands concerning digital certificates and how to import them for use in Knox.

---------------------------------------------------------------------------------------------------------

CA Certificate Steps for Dev and Prod SSL with Apache Knox

The Hortonworks documentation for the CA Cert steps is here.

This article gives you the commands that the doc is asking you to accomplish. This is the heart of this article.

First Some handy Knox specifics:

1. {GATEWAY_HOME}/data/security/keystores/gateway.jks

   This is the identity keystore for the Knox Gateway and needs the public and private keys as well as any signing certs. (see apache docs) The expected alias for the certificate is gateway-identity.

   (Ancil's Note: {GATEWAY_HOME} is usually /usr/hdp/current/knox-server/)

2. {GATEWAY_HOME}/data/security/keystores/__gateway-credentials.jceks.

   This is the credential store for the gateway itself and you will want to add a credential to this that protects the private key passphrase used when you import the key pair into the identity store. This is done with knoxcli.sh create-alias gateway-identity-passphrase --value {value}.

3. The master secret for the gateway is used as the keystore password and must also be used to import the key pair. If you choose to make the private key passphrase the same as the master secret then you can skip #2 above.

(Source @lmccay)

Here are the steps and commands to run

-----------------------------------------------------------------------

Step 1. Export PKCS12 key

If the Master Key Pair for your gateway node is in PEM format, you need to convert into PCKS12 format. This needs to be done in order to import into the Knox java keystore.

"PEM certificates usually have extentions such as .pem, .crt, .cer, and .key. The PKCS12 or PFX format is a binary format for storing the server certificate, any intermediate certificates, and the private key in one encryptable file. PFX files usually have extensions such as .pfx and .p12" (Ref.https://www.sslshopper.com/ssl-converter.html )

#Execute the command substituting for the right files

openssl pkcs12 -export -in <gateway_node>.crt -inkey <gateway_node>.pem  -out <gateway_node>.p12 -name gateway-identity -certfile <company_ca>.crt -caname <any friendly name>
> Enter passphrase for crt: <your company should provide this>
> Create an Export Key:  <Use Knox master Key>

(Reference https://www.openssl.org/docs/manmaster/apps/pkcs12.html )

-----------------------------------------------------------------------

Step 2. Turn the PCKS12 into JKS (Java Key Store) format

When Knox was initially setup with Ambari, a jks was already created with an identity of "gateway-identity" for evaluation purposes. See http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Knox_Gateway_Admin_Guide/content/se...

However we need to take our PCKS12 key and use that instead in the Gateway java keystore.

#In Ambari, shut down Knox

#Create a copy or backup of /usr/hdp/current/knox-server/data/security/keystores/gateway.jks
mv  /usr/hdp/current/knox-server/data/security/keystores/gateway.jks  /usr/hdp/current/knox-server/data/security/keystores/gateway.jks.old

#Execute the command substituting for the right files.  IMPORTANT: The Alias MUST be called gateway-identity

keytool -importkeystore -srckeystore <gateway_node>.p12 -srcstoretype pkcs12 -srcstorepass <Knox Master Key> -destkeystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks -deststoretype jks -deststorepass <You can use Knox Master Key or Other> -alias gateway-identity

#Verify that the key was imported
 keytool -list -keystore /usr/hdp/current/knox-server/data/security/keystores/gateway.jks

-----------------------------------------------------------------------

Step 3: Sync Default Gateway Identity

If you did NOT use the Knox Master Key for your destination pass phrase in Step 3 above you need to let Knox know.

knoxcli.sh create-alias gateway-identity-passphrase --value {value}.

If you did use the Knox Master Key for you destination pass phrase in Step 3 above, delete the default credential and when Knox restarts it will automatically create the credential with the Knox Master Key.

knoxcli.sh delete-alias gateway-identity-passphrase

NOTE:

To change password for gateway.jks

keytool -storepasswd -new <master key> -keystore /var/lib/knox/data-2.3.4.0-3485/security/keystores/gateway.jks

-----------------------------------------------------------------------

Step 4: You many need to import your Company's Digital CA Certificate into the Java Key Store

keytool -importcert -trustcacerts -file <company_ca>.crt -storepass changeit -noprompt -alias MyLdapCert -keystore ${JAVA_HOME}/jre/lib/security/cacerts

Enter keystore password:#the default is 'changeit' unless someone else changed it :-)
>changeit

-----------------------------------------------------------------------

Step 5: Start Knox Ambari and Test with appropriate Curl commands.

Set Debug on in Ambari for Knox if you need to do LDAP and SSL connectivity.

Example:

http://<host>:8444/gateway/default/webhdfs/v1/?op=liststatus

curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/default/webhdfs/v1?op=LISTSTATUS'

[#2]

@Pardeep feel free to add or edit

[#3]

Thank you. Hortonworks doc is very scarce about this. Never would have I guessed such commands without your article. Awesome!