question Re: [HDFS] Block replication dfs.replication affect performance in Support Questions

[HDFS] Block replication dfs.replication affect performance

hoangletrung — Mon, 09 Jan 2017 12:41:40 GMT

I have two question about dfs.replication parameter:

1. I know default of replication block is 3. But when I configure dfs.replication=1, Do it affected to cluster performance.

2. I have a lot of data with configure dfs.replication=1, and now I change configure to dfs.replication= 3. So my data will auto replicate or I have to build my data again to replication running. I need to be sure because my data is very important.

P/S: any best practice for dfs.replication configure.

Re: [HDFS] Block replication dfs.replication affect performance

mqureshi — Mon, 09 Jan 2017 13:53:12 GMT

@Hoang Le

1. I know default of replication block is 3. But when I configure dfs.replication=1, Do it affected to cluster performance.

Since you are not replicating, your writes will be faster at the expense of significant risk of data loss as well as read performance. Your reads can be slow because your data might happen to be on a node experiencing issues with no other block available as well as job failure in case of just one node failure.

Use setrep to change replication factor for existing files. It will replicate existing data (you will have to provide the path).

hadoop fs -setrep [-R] [-w] <numReplicas> <path>

hadoop fs -setrep -w 3 /user/hadoop/dir1

The -R flag is accepted for backwards compatibility. It has no effect.

The -w flag requests that the command wait for the replication to complete. This can potentially take a very long time.
Returns 0 on success and -1 on error.

P/S: any best practice for dfs.replication configure.

Always use default replication factor of 3. It provides data resiliency as well as redundancy in case of node failures. It also helps read performance. In rare cases, you can increase replication factor to help even more data distribution to make reads faster.

Re: [HDFS] Block replication dfs.replication affect performance

hoangletrung — Mon, 19 Aug 2019 10:41:43 GMT

@mqureshi

Thank you for your answers.

I want ask one more question.

If I change just only on Ambari UI. So Is it equal with I used setrep command ? Or I need configure on Ambari UI before use setrep

Re: [HDFS] Block replication dfs.replication affect performance

mqureshi — Mon, 09 Jan 2017 14:10:20 GMT

@Hoang Le

No, Ambari UI will set it for future files that you will create. It will not run setrep command for you. That you will have to run from shell as described above.