Enable HTTPS for HDFS
To enable HTTPS for web HDFS, do the following:
Step 1:
Get the keystore to use in HDFS configurations.
a) In case cert is getting signed by CA, do the following:
1. Generate a keystore for each host. Make sure the common name portion of the certificate matches the hostname where the certificate will be deployed.
keytool -genkey -keyalg RSA -alias c6401 -keystore /tmp/keystore.jks -storepass bigdata -validity 360 -keysize 2048
2. Generate CSR from above keystore
keytool -certreq -alias c6401 -keyalg RSA -file /tmp/c6401.csr -keystore /tmp/keystore.jks -storepass bigdata
3. Now get the singed cert from CA - file name is /tmp/c6401.crt
4. Import the root cert to JKS first. (Ignore if it already present)
keytool -import -alias root -file /tmp/ca.crt -keystore /tmp/keystore.jks
Note: here ca.crt is root cert
5. Repeat step4 for intermediate cert if there is any.
6. Import signed cert into JKS
keytool -import -alias c6401 -file /tmp/c6401.crt -keystore /tmp/keystore.jks -storepass bigdata
7. Import root cert to trust store (Here it creates new truststore.jks )
keytool -import -alias root -file /tmp/ca.crt -keystore /tmp/truststore.jks -storepass bigdata
8. Import intermediate cert (if there is any) to trust store (similar to step 7)
OR,
b) Do the following steps in case you are planning to use self-signed cert.
1. Generate a keystore for each host. Make sure the common name portion of the certificate matches the hostname where the certificate will be deployed.
# keytool -genkey -keyalg RSA -alias c6401 -keystore /tmp/keystore.jks -storepass bigdata -validity 360 -keysize 2048
2. Generate truststore
Note: Truststore must contains certificate of all servers, you can use below commands to export cert from keystore and then import it to truststore
# keytool -export -file /tmp/c6401.crt -keystore /tmp/truststore.jks -storepass bigdata -alias c6401 -rfc
# keytool -import -alias c6401 -file /tmp/c6401.crt -keystore /tmp/truststore.jks -storepass bigdata
Step 2:
Import truststore certificates to java truststore (cacerts or jssecacerts)
keytool -importkeystore \
-srckeystore /tmp/truststore.jks \
-destkeystore /usr/java/default/jre/lib/security/cacerts \
-deststorepass changeit \
-srcstorepass bigdata
Step 3:
Login to Ambari and configure/ add following properties in core-site.xml.
hadoop.ssl.require.client.cert=false
hadoop.ssl.hostname.verifier=DEFAULT
hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory
hadoop.ssl.server.conf=ssl-server.xml
hadoop.ssl.client.conf=ssl-client.xml
Step 4:
Add/ modify following properties in hdfs-site.xml:
For non-HA cluster:
dfs.http.policy=HTTPS_ONLY
dfs.client.https.need-auth=false
dfs.datanode.https.address=0.0.0.0:50475
dfs.namenode.https-address=NN:50470
dfs.namenode.secondary.https-address=c6401-node3.coelab.cloudera.com:50091
Note: you can also set dfs.http.policy=HTTP_AND_HTTPS
For HA-enabled clusters:
dfs.http.policy=HTTPS_ONLY
dfs.client.https.need-auth=false
dfs.datanode.https.address=0.0.0.0:50475
dfs.namenode.https-address.<nameservie>.nn1= c6401-node2.coelab.cloudera.com:50470
dfs.namenode.https-address.<nameservie>.nn2= c6401-node3.coelab.cloudera.com:50470
dfs.journalnode.https-address=0.0.0.0:8481
Step 5:
Update the following configurations under Advanced ssl-server (ssl-server.xml)
ssl.server.truststore.location=/tmp/truststore.jks
ssl.server.truststore.password=bigdata
ssl.server.truststore.type=jks
ssl.server.keystore.location=/tmp/keystore.jks
ssl.server.keystore.password=bigdata
ssl.server.keystore.keypassword=bigdata
ssl.server.keystore.type=jks
Step 6:
Update the following configurations under Advanced ssl-client (ssl-client.xml)
ssl.client.truststore.location=/tmp/truststore.jks
ssl.client.truststore.password=bigdata
ssl.client.truststore.type=jks
ssl.client.keystore.location=/tmp/keystore.jks
ssl.client.keystore.password=bigdata
ssl.client.keystore.keypassword=bigdata
ssl.client.keystore.type=jks
Step 7:
Restart HDFS service
Step 8:
Import the CA root (and Intermediate, if any) to ambari-server truststore by running:
ambari-server setup-security
For self-signed certs, make sure you import namenode(s) certificates to ambari-server truststore
Refer to Steps to set up Truststore for Ambari Server for more details.
Step 9:
Open namenode web UI in https mode on 50470 port
Tips:
- When you enable the HTTPS for HDFS, Journal node and NN starts in HTTPS mode; check for journal node and namenode logs for any errors.
- You can skip the step to create truststore.jks file and make use to java truststore instead. However, ensure you import certs (all required certs) to java truststore.
More articles
