Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]

Cloudera Community

Community Articles

Find and share helpful community-sourced technical articles.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.
Hi, I'm CLEO! Something exciting is coming to the Community. Stay Tuned!

How to transfer file using secure webhdfs in distcp.

Labels (1)
Labels:
avatar
author-rank mfang
New Member

Created on ‎03-31-2017 06:16 PM

Cluster setup :

Client hostnames : mf-amos110.novalocal , and mf-amos111.novalocal
Server hostnames : node1.openstacklocal , node2.openstacklocal and node3.openstacklocal
step 1. create Company CA "rootCA.key" , on host mf-amos111.novalocal : 
[root@mf-amos111 centos]# openssl genrsa -aes256 -out rootCA.key 4096
Step 2 . create a pem file "rootCA.pem", which will be used to sign certificates sent by other hosts :
[root@mf-amos111 centos]# openssl req -x509 -new -key rootCA.key -days 1095 -out rootCA.pem
Step 3. Create a truststore file "all-truststore.jks" for all hosts, and copy this keystore to /etc/hadoop/conf dir on each host.
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore all-truststore.jks -alias CARoot -import -file rootCA.pem

[root@mf-amos111 centos]# cp all-truststore.jks /etc/hadoop/conf

[root@mf-amos111 conf]# scp all-truststore.jks [email protected]:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]# scp all-truststore.jks root@node1:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]# scp all-truststore.jks root@node2:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]# scp all-truststore.jks root@node3:/etc/hadoop/conf
all-truststore.jks                                                                                                         100% 1506     1.5KB/s   00:00    
[root@mf-amos111 conf]#
Step 4. Create private keystore on each host in both clusters :
[root@node1 centos]# keytool -genkey -alias node1 -keyalg RSA -keystore node1-keystore.jks -keysize 2048
 ....
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -genkey -alias amos111 -keyalg RSA -keystore amos111-keystore.jks -keysize 2048
NOTE : make sure the answer for "What is your first and last name?" is each of your host's FQDN !! Step 5. Generate a request to sign file ".csr" on each host 
[root@node1 centos]# keytool -certreq -alias node1 -keystore node1-keystore.jks -file node1-keystore.csr
 ....
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -certreq -alias amos111 -keystore amos111-keystore.jks -file amos111-keystore.csr
Step 6: scp the csr file to mf-amos111.novalocal , to get them signed by rootCA 
[root@node1 centos]# scp node1-keystore.csr [email protected]:/home/centos
 ...
[root@mf-amos110 centos]# scp amos110-keystore.csr [email protected]:/home/centos
Step 7 : sign the csr files on mf-amos111.novalocal , and get a crt file for each host 
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in amos111-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out amos111-keystore.crt -days 730
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=mf-amos111.novalocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in amos110-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out amos110-keystore.crt -days 730
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=mf-amos110.novalocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in node1-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node1-keystore.crt -days 730 
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node1.openstacklocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in node2-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node2-keystore.crt -days 730 
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node2.openstacklocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]# openssl x509 -sha256 -req -in node3-keystore.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out node3-keystore.crt -days 730 
Signature ok
subject=/C=US/ST=CA/L=SantaClara/O=Hortonworks/OU=Support/CN=node3.openstacklocal
Getting CA Private Key
Enter pass phrase for rootCA.key:
[root@mf-amos111 centos]#
Step 8 : Copy the crt file, and the rootCA.pem file to each host in both clusters :
[root@mf-amos111 centos]# scp rootCA.pem [email protected]:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp rootCA.pem root@node1:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp rootCA.pem root@node2:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp rootCA.pem root@node3:/home/centos
rootCA.pem                                                                                                                 100% 2009     2.0KB/s   00:00    
[root@mf-amos111 centos]# scp amos110-keystore.crt [email protected]:/home/centos
amos110-keystore.crt                                                                                                       100% 1566     1.5KB/s   00:00    
[root@mf-amos111 centos]# scp node1-keystore.crt root@node1:/home/centos
node1-keystore.crt                                                                                                         100% 1566     1.5KB/s   00:00    
[root@mf-amos111 centos]# scp node2-keystore.crt root@node2:/home/centos
node2-keystore.crt                                                                                                         100% 1566     1.5KB/s   00:00    
[root@mf-amos111 centos]# scp node3-keystore.crt root@node3:/home/centos
node3-keystore.crt                                                                                                         100% 1566     1.5KB/s   00:00
Step 9 : Import the crt file, and the rootCA.pem to the keystore files on each host 
[root@node1 centos]# keytool -keystore node1-keystore.jks -alias rootCA -import -file rootCA.pem
Enter keystore password:  
Owner: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Issuer: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Serial number: fbe9c20f767c716d
Valid from: Thu Mar 09 00:38:14 UTC 2017 until: Sun Mar 08 00:38:14 UTC 2020
Certificate fingerprints:
MD5:  62:A4:AE:D3:56:40:06:D3:80:25:3E:10:CB:18:81:CD
SHA1: E6:A3:67:03:EC:3D:2F:49:F5:94:E6:D9:CC:0B:CD:0A:F1:0E:E9:30
SHA256: 95:63:B6:9D:BD:EC:BA:34:23:3D:B1:06:98:A2:CF:0C:61:F9:6D:ED:79:77:17:E7:3D:37:D5:54:FD:74:23:9E
Signature algorithm name: SHA1withRSA
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
Trust this certificate? [no]:  yes
Certificate was added to keystore
[root@node1 centos]# keytool -keystore node1-keystore.jks -alias node1 -import -file node1-keystore.crt
Enter keystore password:  
Certificate reply was installed in keystore
.....

.....

[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore amos111-keystore.jks -alias rootCA -import -file rootCA.pem
Enter keystore password:  
Owner: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Issuer: CN=MyCA, OU=Support, O=Hortonworks, L=SantaClara, ST=CA, C=US
Serial number: fbe9c20f767c716d
Valid from: Thu Mar 09 00:38:14 UTC 2017 until: Sun Mar 08 00:38:14 UTC 2020
Certificate fingerprints:
MD5:  62:A4:AE:D3:56:40:06:D3:80:25:3E:10:CB:18:81:CD
SHA1: E6:A3:67:03:EC:3D:2F:49:F5:94:E6:D9:CC:0B:CD:0A:F1:0E:E9:30
SHA256: 95:63:B6:9D:BD:EC:BA:34:23:3D:B1:06:98:A2:CF:0C:61:F9:6D:ED:79:77:17:E7:3D:37:D5:54:FD:74:23:9E
Signature algorithm name: SHA1withRSA
Version: 3
Extensions: 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 78 8D 63 37 DF 19 BE C5   BC 42 36 09 57 FD 70 FF  x.c7.....B6.W.p.
0010: C4 AB 76 D2                                        ..v.
]
]
Trust this certificate? [no]:  yes
Certificate was added to keystore
[root@mf-amos111 centos]# /usr/jdk64/jdk1.8.0_77/jre/bin/keytool -keystore amos111-keystore.jks -alias amos111 -import -file amos111-keystore.crt
Enter keystore password:  
Certificate reply was installed in keystore
Step 10 : Copy the latest keystore files to /etc/hadoop/conf dir as well, 
[root@node1 centos]# cp /home/centos/node1-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@node2 centos]# cp /home/centos/node2-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@node3 centos]# cp /home/centos/node3-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@mf-amos110 centos]# cp /home/centos/amos110-keystore.jks /etc/hadoop/conf/all-keystore.jks

[root@mf-amos111 conf]# cp /home/centos/amos111-keystore.jks /etc/hadoop/conf/all-keystore.jks
Step 11 : configure following properties in both Ambari -> hdfs config for both clusters.
Advanced ssl-client  -> ssl.client.keystore.location -> /etc/hadoop/conf/all-keystore.jks

Advanced ssl-client  ->  ssl.client.truststore.location -> /etc/hadoop/conf/all-truststore.jks
change the passwords accordingly 
Advanced ssl-server ->  ssl.server.truststore.location -> /etc/hadoop/conf/all-truststore.jks

Advanced ssl-server -> ssl.server.keystore.location -> /etc/hadoop/conf/all-keystore.jks
change the passwords accordingly
Advanced hdfs-site -> dfs.http.policy -> HTTP_AND_HTTPS
Step 12 : restart both HDFS Step 13 : test 
[hdfs@mf-amos111 ~]$ hadoop distcp swebhdfs://mf-amos110.novalocal/tmp/id1aacbb60_date310917 swebhdfs://node1.openstacklocal/tmp/
17/03/09 23:23:31 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=false, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[swebhdfs://mf-amos110.novalocal/tmp/id1aacbb60_date310917], targetPath=swebhdfs://node1.openstacklocal/tmp, targetPathExists=true, filtersFile='null'}
17/03/09 23:23:31 INFO impl.TimelineClientImpl: Timeline service address: http://mf-amos111.novalocal:8188/ws/v1/timeline/
17/03/09 23:23:31 INFO client.RMProxy: Connecting to ResourceManager at mf-amos111.novalocal/172.26.96.188:8050
17/03/09 23:23:31 INFO client.AHSProxy: Connecting to Application History server at mf-amos111.novalocal/172.26.96.188:10200
17/03/09 23:23:33 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0
17/03/09 23:23:33 INFO tools.SimpleCopyListing: Build file listing completed.
17/03/09 23:23:33 INFO tools.DistCp: Number of paths in the copy list: 1
17/03/09 23:23:33 INFO tools.DistCp: Number of paths in the copy list: 1
17/03/09 23:23:33 INFO impl.TimelineClientImpl: Timeline service address: http://mf-amos111.novalocal:8188/ws/v1/timeline/
17/03/09 23:23:33 INFO client.RMProxy: Connecting to ResourceManager at mf-amos111.novalocal/172.26.96.188:8050
17/03/09 23:23:33 INFO client.AHSProxy: Connecting to Application History server at mf-amos111.novalocal/172.26.96.188:10200
17/03/09 23:23:34 INFO mapreduce.JobSubmitter: number of splits:1
17/03/09 23:23:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489078447425_0004
17/03/09 23:23:34 INFO impl.YarnClientImpl: Submitted application application_1489078447425_0004
17/03/09 23:23:34 INFO mapreduce.Job: The url to track the job: http://mf-amos111.novalocal:8088/proxy/application_1489078447425_0004/
17/03/09 23:23:34 INFO tools.DistCp: DistCp job-id: job_1489078447425_0004
17/03/09 23:23:34 INFO mapreduce.Job: Running job: job_1489078447425_0004
17/03/09 23:23:42 INFO mapreduce.Job: Job job_1489078447425_0004 running in uber mode : false
17/03/09 23:23:42 INFO mapreduce.Job:  map 0% reduce 0%
17/03/09 23:23:50 INFO mapreduce.Job:  map 100% reduce 0%
17/03/09 23:23:50 INFO mapreduce.Job: Job job_1489078447425_0004 completed successfully
17/03/09 23:23:50 INFO mapreduce.Job: Counters: 38
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=143285
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=375
HDFS: Number of bytes written=0
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
SWEBHDFS: Number of bytes read=1309
SWEBHDFS: Number of bytes written=1309
SWEBHDFS: Number of read operations=11
SWEBHDFS: Number of large read operations=0
SWEBHDFS: Number of write operations=2
Job Counters 
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=5098
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5098
Total vcore-milliseconds taken by all map tasks=5098
Total megabyte-milliseconds taken by all map tasks=5220352
Map-Reduce Framework
Map input records=1
Map output records=0
Input split bytes=115
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=91
CPU time spent (ms)=3190
Physical memory (bytes) snapshot=252891136
Virtual memory (bytes) snapshot=2787827712
Total committed heap usage (bytes)=113770496
File Input Format Counters 
Bytes Read=260
File Output Format Counters 
Bytes Written=0
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESCOPIED=1309
BYTESEXPECTED=1309
COPY=1
[hdfs@mf-amos111 ~]$
4,006 Views
Announcements
Community Announcements
Community Upgrade Announcement
What's New @ Cloudera
Announcing Cloudera Flow Management Operator for Kubernetes ...
What's New @ Cloudera
Announcing Cloudera Streams Messaging Operator for Kubernete...
Community Announcements
July 2026 Community Highlights
Community Announcements
June 2026 Community Highlights
View More Announcements
Version history
Last update:
‎03-31-2017 06:16 PM
Updated by:
New Member
Contributors