hadoop distcp command failing

[#1]

distcp-error.txtthis command is failing and I am unable to find the reason looking at the log file ,please help identify the issue, log file attached.

hadoop distcp hdfs:///user/sami/ hdfs:///user/zhang

[#2]

@Sami Ahmad

The following line seems to indicate the issue:

Caused by: java.io.IOException: Check-sum mismatch between hdfs://hadoop1.tolls.dot.state.fl.us:8020/user/sami/error1.log and hdfs://hadoop1.tolls.dot.state.fl.us:8020/user/zhang/.distcp.tmp.attempt_1472051594557_0001_m_000001_0. Source and target differ in block-size. Use -pb to preserve block-sizes during copy. Alternatively, skip checksum-checks altogether, using -skipCrc. (NOTE: By skipping checksums, one runs the risk of masking data-corruption during file-transfer.)

Is the block size set differently between the source and target clusters?

[#3]

the source and target clusters? I am using same node ..hadoop1 . so I guess the block size would be same.

how can I check all this?

[#4]

so you have two clusters on same node? Is it possible that two clusters have different block size settings? Can you please verify dfs.blocksize setting on both clusters?

[#5]

@Sami Ahmad

Try below command

hadoop distcp -cp hdfs:///user/sami/ hdfs:///user/zhang

[#6]

wrong syntax . "-cp doesn't exist"

[#7]

@Sami Ahmad

Below is the correct syntax

hadoop distcp -pc hdfs:///user/sami/ hdfs:///user/zhang

[#8]

@Sami Ahmad

If i'm not wrong , you are trying to copy the data within same cluster to different directories.

You can simply use the copy command.

hadoop fs -cp hdfs:///user/sami/ hdfs:///user/zhang

[#9]

I want to use distcp for learning purposes.

[#10]

@Sami Ahmad

Can you try distcp2 instead?

hadoop distcp2 hdfs:///user/sami/ hdfs:///user/zhang