SSL configuration for Distcp accross the cluster in wire encrypted Multicluster Envrionment

[#1]

this Article explain the additional steps required to configure wire encryption by exporting/importing the certificates across the cluster for distcp to work on wire encrypted multi cluster .

Problem: on wire encrypted multi cluster environment distcp fails if steps given in this article are not performed, we may see ssl error as follows:

javax.net.ssl.SSLHandshakeException: DestHost:destPort <KMS_HOST>:9393 , LocalHost:localPort null:0. Failed on local exception: javax.net.ssl.SSLHandshakeException: Error while authenticating with endpoint: https://<KMS_HOST>e:9393/kms/v1/?op=GETDELEGATIONTOKEN&rene.
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) 

Prerequisites:

1) two cluster should be setup with Ranger

2) wire encryption should be enabled on the both clusters already. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/content/enabling-ssl-for-compon...

3) if Ranger kms is installed then wire encryption should be enabled for ranger kms too in both the clusters.

steps to configure SSL for distcp to work in multi cluster:

1) export the certificate from Hadoop server key store file on all the host part of the cluster1 and cluster2.

cd <server_hadoop_key_location>;keytool -export -alias hadoop_cert_<host_name> -keystore <keystore_file_path> -rfc -file hadoop_cert_<host_name> -storepass <keystore_password> 

Note: if you don't know the location of the key store, then you can search for config "ssl.server.keystore.location" in the hdfs config

2) copy all the certificates generated for cluster1 in previous step from cluster1 hosts to client key location on all the hosts part of cluster2. and similarly copy all the certificates generated for cluster2, from cluster2 hosts to client key location on all the host part of cluster1

3) Import the all the cluster1 certificates to the hadoop client trustore on all the host of cluster2 and vice versa.

   cd <client_hadoop_key_location>;keytool import -noprompt -alias hadoop_cert_<host_name> -file hadoop_cert_<host_name> -keystore <truststore_file_path> -storepass <truststore_password> 

Note: if you don't know the location of the truststore, then you can search for config "ssl.client.truststore.location" in the hdfs config

Additional steps if Ranger Kms is installed:

if ranger kms is installed then we need to export the ranger kms certificate from ranger kms hosts of cluster1 to Hadoop client trust store of cluster2

1) export the certificate from Ranger kms server key store file on kms hosts part of the cluster1 and cluster2.

cd <kms_key_store_location>;keytool -export -alias kms_cert_<host_name> -keystore <kms_keystore_file_path> -rfc -file kms_cert_<host_name> -storepass <kms_keystore_password> 

Note: if you don't know the location of the kms key store, then you can search for config "ranger.https.attrib.keystore.file" in the kms config

2) copy all the certificates generated for kms in cluster1 in previous step from cluster1 kms hosts to client key location on all the hosts part of cluster2. and similarly copy all the certificates generated for kms in cluster2, from cluster2 kms hosts to client key location on all the host part of cluster1

3) Import all the cluster1 kms certificates to the Hadoop client trust store on all the host of cluster2 and vice versa.

   cd <client_hadoop_key_location>;keytool import -noprompt -alias kms_cert_<host_name> -file kms_cert_<host_name> -keystore <truststore_file_path> -storepass <truststore_password> 

Now restart Hdfs, Yarn, Mapreduce and Ranger KMS on both the cluster and once both the services successfully started, try distcp it should work fine.

hadoop distcp -Dmapreduce.job.hdfs-servers.token-renewal.exclude=cluster1 -skipcrccheck -update /distcp_cluster1 hdfs://cluster2/distcp_cluster2/