How can I fixed this?
/tmp/hadoop-yarn/staging/ubuntu/.staging/job_1450038005671_0025/job.jar: Under replicated BP-938294433-10.0.1.190-1450037861153:blk_1073744219_3398. Target Replicas is 10 but found 3 replica(s).
I get this when I run "hadoop fsck /" in my master node.
Assumption: You are running setrep command. I am assuming that dfs.replication is set to 3. I am guessing that you are setting the replication 10 for some files.
You are running fsck while setrep is running.
[hdfs@phdns02 ~]$ hdfs dfs -setrep -w 10 /tmp
Replication 10 set: /tmp/bat/BattingPost.csv
Replication 10 set: /tmp/dummy.tar.gz
Waiting for /tmp/bat/BattingPost.csv ...............
[hdfs@phdns01 ~]$ hadoop fsck /tmp
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Connecting to namenode via xxxx
FSCK started by hdfs (auth:SIMPLE) from /172.24.64.98 for path /tmp at Sun Dec 13 16:05:00 PST 2015
/tmp/bat/BattingPost.csv: Under replicated BP-301214548-172.24.64.98-1447961741360:blk_1073743585_2761. Target Replicas is 10 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
/tmp/dummy.tar.gz: Under replicated BP-301214548-172.24.64.98-1447961741360:blk_1073743218_2394. Target Replicas is 10 but found 2 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
Total size:621875 B
Total blocks (validated):2 (avg. block size 310937 B)
Minimally replicated blocks:2 (100.0 %)
Over-replicated blocks:0 (0.0 %)
Under-replicated blocks:2 (100.0 %)
Mis-replicated blocks:0 (0.0 %)
Default replication factor:1
Average block replication:2.0
Missing replicas:16 (80.0 %)
Number of data-nodes:2
Number of racks:1
FSCK ended at Sun Dec 13 16:05:00 PST 2015 in 129 milliseconds
The filesystem under path '/tmp' is HEALTHY
Usage: hadoop fs -setrep [-R] [-w] <numReplicas> <path>
Changes the replication factor of a file. If path is a directory then the command recursively changes the replication factor of all files under the directory tree rooted at path.
Returns 0 on success and -1 on error.
The replication level for submitted job files. This should be around the square root of the number of nodes.
With that, and having 1 master nodes and two slaves should I leave it to be 10? If that's the case why I get that message then? What do you suggest me to set it to? I thought I have to set it to 2!
If you want to avoid that underreplicated blocks message at all, then set it to "2".
If you go with the default setting of "10" you will see that message only for job's that fail accidentially. Normally those files you listed in your initial post are being deleted after job execution and you won't see underreplicated blocks.