question Re: Can we fix the under Under replicated blocks while doing the copyToLocal in Archives of Support Questions (Read Only)

Can we fix the under Under replicated blocks while doing the copyToLocal

Iron — Wed, 27 Dec 2017 16:04:27 GMT

Hi Guys,

I am working on a backup task in DEV. I am doing the datanode backup to local. I have set the safemode is OFF and started the data backup. While in the middle of backup, I have noticed there is 1 under replicated blocks and 1 missing block. So i am planning to do the fix. So my doubt is, We can do the fix parallely ?

Re: Can we fix the under Under replicated blocks while doing the copyToLocal

saranvisa — Wed, 27 Dec 2017 17:16:53 GMT

@Iron

In my opinion both are two different tasks, and it is upto you to do it in parallel

also the copyToLocal should work even with under replicated block as only one replication out of three (if you follow default 3 replication) will be copied to local

also not sure i am missing something here, but wondering why you are using copyToLocal option for backup instead of Cloudera Manager -> Backup (menu) -> Replication schedule -> Create schedule (pre-request: set the peer) option, it may reduce your work

Re: Can we fix the under Under replicated blocks while doing the copyToLocal

Iron — Thu, 28 Dec 2017 07:16:18 GMT

Currently we dont have cloudera manager. so any other alternate method other than copyToLocal?. and also we have mandatory to enable the Safemode on for doing the copyTolocal?

Re: Can we fix the under Under replicated blocks while doing the copyToLocal

saranvisa — Thu, 28 Dec 2017 16:08:03 GMT

@Iron

I don't think it is mandatory to enable the safemode during copyToLocal. May be you can use safemode to make sure nobody is updating/deleting/inserting the data during the data copy.

I know the difficulties without cloudera manager/hortonworks, etc

Long back, i've used the below export/import method for Hive table backup, again this will export the data to HDFS and you still have to use copyTolocal. The advantage is, it will also take care of metadata

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport#LanguageManualImportExport-ExportSyntax

You can use these options as temporary solution but once you start using cloudera manager or any other management tool, I would recommend to use the backup option that i've mentioned earlier