Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS mover does not complete in one execution

HDFS mover does not complete in one execution

New Contributor

Hi,

 

I'm using heterogeneous storage, after changing policy for a folder I'm executing:

hdfs mover -p <path>

It is working correctly but after some random number of iterations it stops with no errors. After checking some of the blocks are moved to proper disks but some are still not moved.

Executing the same command few times allows to move all the blocks to correct disks.

 

Do any of you know why this happens? Is it normal behaviour?

I want to have it working in one execution or at least know why this is happening.

 

I tried changing many parameters but still the same. I'm using CDH 5.12.0.

 

Thanks

3 REPLIES 3

Re: HDFS mover does not complete in one execution

New Contributor

Hi, 

 

I have the same problem. Did you solve this problem?

Re: HDFS mover does not complete in one execution

Cloudera Employee

Hi Wiktorio,

 

this topic came up when I searched for the same issue, it seems to be a bug that was introduced with a change in CDH5.10.0 and above versions due to HDFS-11015.

 

I am working on a fix under HDFS-13174, is it possible that you experienced the same issue as described there?

 

Istvan

Re: HDFS mover does not complete in one execution

New Contributor

Hi Istvan,

 

This is not the same issue. I didn't receive any error, after some iterations it just stopped.

 

I went through the source code and found out that this is a bug (or a feature).

 

As a workaroud I managed to write a script that executed mover in a loop until data was
migrated completely.

 

Regards,

Wiktor