Hello, I have a cluster with HDP 2.5.0.2 (HDFS version 2.7.1.2.5) and I'm trying to distcp large file (200G) from on-premise cluster to Netapp S3, using fast upload. a map task is launched for each file and each map task reads something like this Task:
task_1518158406102_3329_m_000000
Status:
(84.5% Copying hdfs://HDP1/user/backup/backup-test/2tb_of_200M_each/test_io_0 to s3a://hadoop-acceptance-tests/user/backup/backup-test/test_io_0 [169.1G/200.0G] > map) Once the progress reaches 100% (200.0G/200.0G), then it starts again from 0% (0%/200.0G). This repeats for few times. Here is the command that I used to trigger distcp hadoop distcp -Dfs.s3a.endpoint=s3.in.myhost:8082 -Dfs.s3a.access.key=XXXXXXXXX -Dfs.s3a.secret.key=YYYYYYYYYYY -Dfs.s3a.signing-algorithm=S3SignerType \
-Dfs.s3a.buffer.dir=<Local_path>/tmp-hadoop_backup \
-Dfs.s3a.fast.upload=true \
-Dfs.s3a.fast.buffer.size=1048576 \
-Dfs.s3a.multipart.size=10485760 \
-Dfs.s3a.multipart.threshold=10485760 \
-Dmapreduce.map.memory.mb=8192 \
-Dmapreduce.map.java.opts=-Xmx7360m \
-log /backup/log/distcp_2tb \
-m=300 -bandwidth 1024 \
-skipcrccheck \
-update hdfs:/user/backup/load-test/2tb_of_200M_each s3a://hadoop-acceptance-tests/user/backup/load-test/
Q1. What is the meaning of running each task several times? Q2. After running the job for several hours, job fails with not even copying a single file. It worked (copied) for 10G, 50G and 100G file size each. Q3. Any recommendations/ thumb rules to change/tune the configs based on input file size and upload speeds to s3? Thanks Surya
... View more
