Ranger and HDFS over SSL

[#1]

I am trying to set up SSL between Ranger Admin and Ranger HDFS plugin (HDP 2.6.5)

I am able to successfully have Ranger Admin UI served via HTTPS (browser says certificate is valid), but my HDFS plugin is not able to sync the policies with Ranger.

My setup:

4-node cluster

• Ranger Admin on Node 1
• HDFS NameNode on Node 1
• HDFS SNameNode on Node 2
• HDFS DataNode on Node 4

As far as I understand, I need to set up keystores and truststores on Node 1 and configure Ranger & HDFS to use those keystores and truststores.

Steps performed:

1. Obtained the PKCS7b (.pem) certificate file from my CA (my CA only offers DER, PKCS7b and CRT files)
2. Created a keystore for Ranger Admin Service from the certificate

cp cert.pem cert.p7b

openssl pkcs7 -print_certs -in cert.p7b -out cert.cer

openssl pkcs12 -export -in cert.cer -inkey certkey.key -out rangeradmin.p12 -name rangeradmin

keytool -importkeystore -deststorepass pass2word -destkeypass pass2word -srckeystore rangeradmin.p12 -srcstoretype PKCS12 -destkeystore ranger-admin-keystore.jks -deststoretype JKS -alias rangeradmin

3. Similarly, created a keystore for HDFS

keytool -importkeystore -deststorepass pass2word -destkeypass pass2word -srckeystore rangeradmin.p12 -srcstoretype PKCS12 -destkeystore hdfs-plugin-keystore.jks -deststoretype JKS -alias hdfsplugin	

4. Created Ranger Admin truststore

keytool -export -keystore hdfs-plugin-keystore.jks -alias hdfsplugin -file hdfsplugin.cer -storepass pass2word

keytool -import -file hdfsplugin.cer -alias hdfsplugin -keystore /etc/ranger/admin/conf/ranger-admin-truststore.jks -storepass pass2word

5. Similarly, created HDFS truststore

   keytool -export -keystore ranger-admin-keystore.jks -alias rangeradmin -file rangeradmin.cer -storepass pass2word
   

   keytool -import -file rangeradmin.cer -alias rangeradmin -keystore /etc/hadoop/conf/hdfs-plugin-truststore.jks -storepass pass2word
   

   At this point I have my Ranger keystore and truststore files in /etc/ranger/admin/conf and HDFS has its in /etc/hadoop/conf/

6. Set up Ranger to use SSL using Ranger Admin keystore by following https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_security/content/configure_ambari_ranger... . At this point Ranger Admin UI is served via HTTPS (browser says certificate is valid).
7. Set up HDFS to use SSL via Ambari

1. HDFS > Configs > Advanced > ranger-hdfs-policymgr-ssl and set the following properties:

xasecure.policymgr.clientssl.keystore = /etc/hadoop/conf/hdfs-plugin-keystore.jks
xasecure.policymgr.clientssl.keystore.password = pass2word
xasecure.policymgr.clientssl.truststore = hdfs-plugin-truststore.jks
xasecure.policymgr.clientssl.truststore.password = pass2word
HDFS > Configs > Advanced > Advanced ranger-hdfs-plugin-properties
		
 common.name.for.certificate = hdfsplugin

	The problem is that the HDFS Ranger policies do not get synced. Ranger admin'sxa_portal.log says "Unauthorized access. Unable to get client certificate."
		

			What am I missing here?
		

[#2 · Accepted solution]

Hi @Maxim Neaga.
Just to make sure, if you run the following command for every keystore/trustore, are you able to see the certificate?

keytool -v -list -keystore <pathtokeystore>/<keystore/trustore.jks>

It should show up something like this:

[root@vmurakami-1 ~]# keytool -v -list -keystore windows.jks 
Enter keystore password:  
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries
Alias name: nifi-cert
Creation date: Sep 1, 2018
Entry type: trustedCertEntry
Owner: CN=vmurakami-3, OU=NIFI
Issuer: CN=vmurakami-3, OU=NIFI
Serial number: 1649584f09b00000000
Valid from: Fri Jul 13 21:21:14 UTC 2018 until: Mon Jul 12 21:21:14 UTC 2021
Certificate fingerprints:
	 MD5:  02:BE:7D:37:22:5B:A8:37:F2:F0:02:E0:26:96:E7:54
	 SHA1: 1F:D0:EC:B5:1A:6E:E7:E5:B4:65:71:1B:8A:B3:99:C2:2A:50:28:0D
	 SHA256: 14:1C:40:B9:2E:6C:C4:5F:56:C8:9D:76:31:21:B5:CB:E2:FA:B1:A2:BE:9B:CA:7F:0D:B4:72:1B:32:2A:95:69
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 2D ED D1 1D 7F F6 22   A3 60 39 EF CE AC 09 6E  x-.....".`9....n
0010: CD 51 B9 D3                                        .Q..
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]
#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
  Key_CertSign
  Crl_Sign
]
#5: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 78 2D ED D1 1D 7F F6 22   A3 60 39 EF CE AC 09 6E  x-.....".`9....n
0010: CD 51 B9 D3                                        .Q..
]
]
*******************************************
*******************************************
Alias name: windows
Creation date: Sep 1, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=MSEDGEWIN10, OU=NIFI
Issuer: CN=MSEDGEWIN10, OU=NIFI
Serial number: 27ba96c9
Valid from: Sat Sep 01 07:21:44 UTC 2018 until: Tue Aug 27 07:21:44 UTC 2019
Certificate fingerprints:
	 MD5:  B7:FE:EB:0C:3E:7D:EE:E9:58:54:EC:2B:F4:02:9C:0D
	 SHA1: 3A:9A:DD:05:FF:E8:41:99:C8:8B:D4:84:4C:4A:5E:56:6C:46:15:B0
	 SHA256: 22:CD:A6:CE:9E:F0:B8:A3:A8:6E:25:2E:4D:A2:AB:70:4F:98:36:AC:8C:C0:A0:B6:15:22:E8:27:80:CC:F3:A6
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 3B FE 73 64 EC 9C 91 B6   AC 3D EC 44 9D AF DD 66  ;.sd.....=.D...f
0010: B8 DE 4A F8                                        ..J.
]
]
Certificate[2]:
Owner: CN=vmurakami-3, OU=NIFI
Issuer: CN=vmurakami-3, OU=NIFI
Serial number: 1649584f09b00000000
Valid from: Fri Jul 13 21:21:14 UTC 2018 until: Mon Jul 12 21:21:14 UTC 2021
Certificate fingerprints:
	 MD5:  02:BE:7D:37:22:5B:A8:37:F2:F0:02:E0:26:96:E7:54
	 SHA1: 1F:D0:EC:B5:1A:6E:E7:E5:B4:65:71:1B:8A:B3:99:C2:2A:50:28:0D
	 SHA256: 14:1C:40:B9:2E:6C:C4:5F:56:C8:9D:76:31:21:B5:CB:E2:FA:B1:A2:BE:9B:CA:7F:0D:B4:72:1B:32:2A:95:69
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 2D ED D1 1D 7F F6 22   A3 60 39 EF CE AC 09 6E  x-.....".`9....n
0010: CD 51 B9 D3                                        .Q..
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  clientAuth
  serverAuth
]
#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Non_repudiation
  Key_Encipherment
  Data_Encipherment
  Key_Agreement
  Key_CertSign
  Crl_Sign
]
#5: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 78 2D ED D1 1D 7F F6 22   A3 60 39 EF CE AC 09 6E  x-.....".`9....n
0010: CD 51 B9 D3                                        .Q..
]
]
*******************************************
*******************************************
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore windows.jks -destkeystore windows.jks -deststoretype pkcs12".

Also, make sure of 2 things:
- Give the read permission to the ranger hdfs plugin keystore/trustore

chmod o+r keystore.jks truststore.jks

- Put the same owner (CN=<some_name>) in the
Ranger HDFS plugin > Owner of the certificate in Ambari UI
commonNameForCertificate in the Ranger Plugin for HDFS in Ranger UI

And lastly, to make sure, go to the ranger-admin node into the default java keystore "cacerts" and add the client certificate.

find / -name "cacerts" -type f 
keytool -import -file <your cert file> -alias <your alias> -keystore <the_path_for_the_jdk_listed_above>/cacerts -storepass changeit #default passwor

Hope this helps!

[#3]

Thank you for your answer, @Vinicius Higa Murakami, this was very helpful!

It looks like I was missing the last part, I had to add root and intermediate certificates to the default java keystore "cacerts". I wonder why we need to add the certs to the default java keystore, and not the rangeradmin & client keystores?

[#4]

Hello @Maxim Neaga!
I'm glad that I could help you 🙂
Yeah, make totally sense your question. I'd say because you've 3 truststore's:
- 1 x truststore's for the nifi nodes
- 1 x For the Ranger x Nifi plugin
- 1 x For Ranger Admin (which is the java default cacerts).

Hope this helps!