Target Replicas is 10 but found 3 replica(s) - Under replication

[#1]

How can I fixed this?

/tmp/hadoop-yarn/staging/ubuntu/.staging/job_1450038005671_0025/job.jar:  Under replicated BP-938294433-10.0.1.190-1450037861153:blk_1073744219_3398. Target Replicas is 10 but found 3 replica(s).

I get this when I run "hadoop fsck /" in my master node.

[#2]

@Mona Jalal

Assumption: You are running setrep command. I am assuming that dfs.replication is set to 3. I am guessing that you are setting the replication 10 for some files.

You are running fsck while setrep is running.

Test:

[hdfs@phdns02 ~]$ hdfs dfs -setrep -w 10 /tmp

Replication 10 set: /tmp/bat/BattingPost.csv

Replication 10 set: /tmp/dummy.tar.gz

Waiting for /tmp/bat/BattingPost.csv ...............

[hdfs@phdns01 ~]$ hadoop fsck /tmp

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

Connecting to namenode via xxxx

FSCK started by hdfs (auth:SIMPLE) from /172.24.64.98 for path /tmp at Sun Dec 13 16:05:00 PST 2015

.

/tmp/bat/BattingPost.csv: Under replicated BP-301214548-172.24.64.98-1447961741360:blk_1073743585_2761. Target Replicas is 10 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

.

/tmp/dummy.tar.gz: Under replicated BP-301214548-172.24.64.98-1447961741360:blk_1073743218_2394. Target Replicas is 10 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

Status: HEALTHY

Total size:621875 B

Total dirs:27

Total files:2

Total symlinks:0

Total blocks (validated):2 (avg. block size 310937 B)

Minimally replicated blocks:2 (100.0 %)

Over-replicated blocks:0 (0.0 %)

Under-replicated blocks:2 (100.0 %)

Mis-replicated blocks:0 (0.0 %)

Default replication factor:1

Average block replication:2.0

Corrupt blocks:0

Missing replicas:16 (80.0 %)

Number of data-nodes:2

Number of racks:1

FSCK ended at Sun Dec 13 16:05:00 PST 2015 in 129 milliseconds

The filesystem under path '/tmp' is HEALTHY

[hdfs@phdns01 ~]$

https://hadoop.apache.org/docs/current/hadoop-proj...

setrep

Usage: hadoop fs -setrep [-R] [-w] <numReplicas> <path>

Changes the replication factor of a file. If path is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path.

Options:

• The -w flag requests that the command wait for the replication to complete. This can potentially take a very long time.
• The -R flag is accepted for backwards compatibility. It has no effect.

Example:

• hadoop fs -setrep -w 3 /user/hadoop/dir1

Exit Code:

Returns 0 on success and -1 on error.

[#3]

setrep-test.pdf

I ran another test with multiple setrep numbers.

You are hitting this https://issues.apache.org/jira/browse/HDFS-6223

[#4]

@Neeraj Sabharwal : what should be the ideal replication to be used with setrep if i have 4 datanodes?

[#5]

@Neeraj Sabharwal so is there any .xml file in conf directory that I wouldn't need to run this everytime after start-all.sh?

1. hadoop fs -setrep -w 2 /

[#6]

@Mona Jalal Are you running hadoop fs -setrep -w 2 / with 2 data nodes?

I run into this issue as soon as I set -w to > # of DN. https://issues.apache.org/jira/browse/HDFS-6223

I have no comments on mapreduce.client.submit.file.replication

[#7]

@Mona Jalal

have you had a look at property "mapreduce.client.submit.file.replication" ?

This sets the replication level for submitted job files, and is by default set to 10 (link)

Regards...

[#8]

@Gerd Koenig @Neeraj Sabharwal so it says:

The replication level for submitted job files. This should be around the square root of the number of nodes.

With that, and having 1 master nodes and two slaves should I leave it to be 10? If that's the case why I get that message then? What do you suggest me to set it to? I thought I have to set it to 2!

[#9]

If you want to avoid that underreplicated blocks message at all, then set it to "2".

If you go with the default setting of "10" you will see that message only for job's that fail accidentially. Normally those files you listed in your initial post are being deleted after job execution and you won't see underreplicated blocks.

[#10]

@Mona Jalal : did you find a solution to your problem? Would be great to accept either one of the answers...

Thanks and regards