Cloudera Community

Community Articles

Find and share helpful community-sourced technical articles.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to transfer file using secure webhdfs in distcp.

Labels (1)
Labels:
avatar
author-rank mfang
Explorer

Created on ‎03-31-2017 06:16 PM

Cluster setup :

Client hostnames : mf-amos110.novalocal , and mf-amos111.novalocal
Server hostnames : node1.openstacklocal , node2.openstacklocal and node3.openstacklocal
step 1. create Company CA "rootCA.key" , on host mf-amos111.novalocal : 
[root@mf-amos111 centos]# openssl genrsa -aes256 -out rootCA.key 4096
Step 2 . create a pem file "rootCA.pem", which will be used to sign certificates sent by other hosts :
[root@mf-amos111 centos]# openssl req -x509 -new -key rootCA.key -days 1095 -out rootCA.pem
Step 3. Create a truststore file "all-truststore.jks" for all hosts, and copy this keystore to /etc/hadoop/conf dir on each host.
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore all-truststore.jks -alias CARoot -import -file rootCA.pem

[root@mf-amos111 centos]# cp all-truststore.jks /etc/hadoop/conf

[root@mf-amos111 conf]# scp all-truststore.jks root@mf-amos110.novalocal:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]# scp all-truststore.jks root@node1:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]# scp all-truststore.jks root@node2:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]# scp all-truststore.jks root@node3:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]#
Step 4. Create private keystore on each host in both clusters :
[root@node1 centos]# keytool -genkey -alias node1 -keyalg RSA -keystore node1-keystore.jks -keysize 2048
 ....
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -genkey -alias amos111 -keyalg RSA -keystore amos111-keystore.jks -keysize 2048
NOTE : make sure the answer for "What is your first and last name?" is each of your host's FQDN !! Step 5. Generate a request to sign file ".csr" on each host 
[root@node1 centos]# keytool -certreq -alias node1 -keystore node1-keystore.jks -file node1-keystore.csr
 ....
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -certreq -alias amos111 -keystore amos111-keystore.jks -file amos111-keystore.csr
Step 6: scp the csr file to mf-amos111.novalocal , to get them signed by rootCA 
[root@node1 centos]# scp node1-keystore.csr root@mf-amos111.novalocal:/home/centos
 ...
[root@mf-amos110 centos]# scp amos110-keystore.csr root@mf-amos111.novalocal:/home/centos
Step 7 : sign the csr files on mf-amos111.novalocal , and get a crt file for each host 
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in amos111-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out amos111-keystore.crt -days 730
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=mf-amos111.novalocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in amos110-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out amos110-keystore.crt -days 730
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=mf-amos110.novalocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in node1-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node1-keystore.crt -days 730 
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node1.openstacklocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in node2-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node2-keystore.crt -days 730 
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node2.openstacklocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in node3-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node3-keystore.crt -days 730 
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node3.openstacklocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]#
Step 8 : Copy the crt file, and the rootCA.pem file to each host in both clusters :
[root@mf-amos111 centos]# scp rootCA.pem root@mf-amos110.novalocal:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp rootCA.pem root@node1:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp rootCA.pem root@node2:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp rootCA.pem root@node3:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp amos110-keystore.crt root@mf-amos110.novalocal:/home/centos
amos110-keystore.crt                                                                                                       100% 1566     1.5KB/s   00:00    
[root@mf-amos111 centos]# scp node1-keystore.crt root@node1:/home/centos
node1-keystore.crt                                                                                                         100% 1566     1.5KB/s   00:00    
[root@mf-amos111 centos]# scp node2-keystore.crt root@node2:/home/centos
node2-keystore.crt                                                                                                         100% 1566     1.5KB/s   00:00    
[root@mf-amos111 centos]# scp node3-keystore.crt root@node3:/home/centos
node3-keystore.crt                                                                                                         100% 1566     1.5KB/s   00:00
Step 9 : Import the crt file, and the rootCA.pem to the keystore files on each host 
[root@node1 centos]# keytool -keystore node1-keystore.jks -alias rootCA -import -file rootCA.pem
Enter keystore password:  
Owner: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Issuer: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Serial number: fbe9c20f767c716d
Valid from: Thu Mar 09 00:38:14 UTC 2017 until: Sun Mar 08 00:38:14 UTC 2020
Certificate fingerprints:
MD5:  62:A4:AE:D3:56:40:06:D3:80:25:3E:10:CB:18:81:CD
SHA1: E6:A3:67:03:EC:3D:2F:49:F5:94:E6:D9:CC:0B:CD:0A:F1:0E:E9:30
SHA256: 95:63:B6:9D:BD:EC:BA:34:23:3D:B1:06:98:A2:CF:0C:61:F9:6D:ED:79:77:17:E7:3D:37:D5:54:FD:74:23:9E
Signature algorithm name: SHA1withRSA
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
Trust this certificate? [no]:  yes
Certificate was added to keystore
[root@node1 centos]# keytool -keystore node1-keystore.jks -alias node1 -import -file node1-keystore.crt
Enter keystore password:  
Certificate reply was installed in keystore
.....

.....

[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore amos111-keystore.jks -alias rootCA -import -file rootCA.pem
Enter keystore password:  
Owner: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Issuer: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Serial number: fbe9c20f767c716d
Valid from: Thu Mar 09 00:38:14 UTC 2017 until: Sun Mar 08 00:38:14 UTC 2020
Certificate fingerprints:
MD5:  62:A4:AE:D3:56:40:06:D3:80:25:3E:10:CB:18:81:CD
SHA1: E6:A3:67:03:EC:3D:2F:49:F5:94:E6:D9:CC:0B:CD:0A:F1:0E:E9:30
SHA256: 95:63:B6:9D:BD:EC:BA:34:23:3D:B1:06:98:A2:CF:0C:61:F9:6D:ED:79:77:17:E7:3D:37:D5:54:FD:74:23:9E
Signature algorithm name: SHA1withRSA
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
Trust this certificate? [no]:  yes
Certificate was added to keystore
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore amos111-keystore.jks -alias amos111 -import -file amos111-keystore.crt
Enter keystore password:  
Certificate reply was installed in keystore
Step 10 : Copy the latest keystore files to /etc/hadoop/conf dir as well, 
[root@node1 centos]# cp /home/centos/node1-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@node2 centos]# cp /home/centos/node2-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@node3 centos]# cp /home/centos/node3-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@mf-amos110 centos]# cp /home/centos/amos110-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@mf-amos111 conf]# cp /home/centos/amos111-keystore.jks /etc/hadoop/conf/all-keystore.jks
Step 11 : configure following properties in both Ambari -> hdfs config for both clusters.
Advanced ssl-client  -> ssl.client.keystore.location -> /etc/hadoop/conf/all-keystore.jks

Advanced ssl-client  ->  ssl.client.truststore.location -> /etc/hadoop/conf/all-truststore.jks
change the passwords accordingly 
Advanced ssl-server ->  ssl.server.truststore.location -> /etc/hadoop/conf/all-truststore.jks

Advanced ssl-server -> ssl.server.keystore.location -> /etc/hadoop/conf/all-keystore.jks
change the passwords accordingly
Advanced hdfs-site -> dfs.http.policy -> HTTP_AND_HTTPS
Step 12 : restart both HDFS Step 13 : test 
[hdfs@mf-amos111 ~]$ hadoop distcp swebhdfs://mf-amos110.novalocal/tmp/id1aacbb60_date310917 swebhdfs://node1.openstacklocal/tmp/
17/03/09 23:23:31 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[swebhdfs://mf-amos110.novalocal/tmp/id1aacbb60_date310917], targetPath=swebhdfs://node1.openstacklocal/tmp, targetPathExists=true, filtersFile='null'}
17/03/09 23:23:31 INFO impl.TimelineClientImpl: Timeline service address: http://mf-amos111.novalocal:8188/ws/v1/timeline/
17/03/09 23:23:31 INFO client.RMProxy: Connecting to ResourceManager at mf-amos111.novalocal/172.26.96.188:8050
17/03/09 23:23:31 INFO client.AHSProxy: Connecting to Application History server at mf-amos111.novalocal/172.26.96.188:10200
17/03/09 23:23:33 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0
17/03/09 23:23:33 INFO tools.SimpleCopyListing: Build file listing completed.
17/03/09 23:23:33 INFO tools.DistCp: Number of paths in the copy list: 1
17/03/09 23:23:33 INFO tools.DistCp: Number of paths in the copy list: 1
17/03/09 23:23:33 INFO impl.TimelineClientImpl: Timeline service address: http://mf-amos111.novalocal:8188/ws/v1/timeline/
17/03/09 23:23:33 INFO client.RMProxy: Connecting to ResourceManager at mf-amos111.novalocal/172.26.96.188:8050
17/03/09 23:23:33 INFO client.AHSProxy: Connecting to Application History server at mf-amos111.novalocal/172.26.96.188:10200
17/03/09 23:23:34 INFO mapreduce.JobSubmitter: number of splits:1
17/03/09 23:23:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489078447425_0004
17/03/09 23:23:34 INFO impl.YarnClientImpl: Submitted application application_1489078447425_0004
17/03/09 23:23:34 INFO mapreduce.Job: The url to track the job: http://mf-amos111.novalocal:8088/proxy/application_1489078447425_0004/
17/03/09 23:23:34 INFO tools.DistCp: DistCp job-id: job_1489078447425_0004
17/03/09 23:23:34 INFO mapreduce.Job: Running job: job_1489078447425_0004
17/03/09 23:23:42 INFO mapreduce.Job: Job job_1489078447425_0004 running in uber mode : false
17/03/09 23:23:42 INFO mapreduce.Job:  map 0% reduce 0%
17/03/09 23:23:50 INFO mapreduce.Job:  map 100% reduce 0%
17/03/09 23:23:50 INFO mapreduce.Job: Job job_1489078447425_0004 completed successfully
17/03/09 23:23:50 INFO mapreduce.Job: Counters: 38
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=143285
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=375
HDFS: Number of bytes written=0
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
SWEBHDFS: Number of bytes read=1309
SWEBHDFS: Number of bytes written=1309
SWEBHDFS: Number of read operations=11
SWEBHDFS: Number of large read operations=0
SWEBHDFS: Number of write operations=2
Job Counters 
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=5098
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5098
Total vcore-milliseconds taken by all map tasks=5098
Total megabyte-milliseconds taken by all map tasks=5220352
Map-Reduce Framework
Map input records=1
Map output records=0
Input split bytes=115
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=91
CPU time spent (ms)=3190
Physical memory (bytes) snapshot=252891136
Virtual memory (bytes) snapshot=2787827712
Total committed heap usage (bytes)=113770496
File Input Format Counters 
Bytes Read=260
File Output Format Counters 
Bytes Written=0
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESCOPIED=1309
BYTESEXPECTED=1309
COPY=1
[hdfs@mf-amos111 ~]$
3,138 Views
Announcements
Community Announcements
October 2024 Community Highlights
What's New @ Cloudera
Accelerate Your AI with the Cloudera AI Inference Service wi...
What's New @ Cloudera
Updates to the HDFS clusters on AWS environments to add supp...
What's New @ Cloudera
Enhancements to the create-database command
Community Announcements
September 2024 Community Highlights
View More Announcements
Version history
Last update:
‎03-31-2017 06:16 PM
Updated by:
Explorer
Contributors