Reply
New Contributor
Posts: 2
Registered: ‎08-02-2017

HDFS mover does not complete in one execution

Hi,

 

I'm using heterogeneous storage, after changing policy for a folder I'm executing:

hdfs mover -p <path>

It is working correctly but after some random number of iterations it stops with no errors. After checking some of the blocks are moved to proper disks but some are still not moved.

Executing the same command few times allows to move all the blocks to correct disks.

 

Do any of you know why this happens? Is it normal behaviour?

I want to have it working in one execution or at least know why this is happening.

 

I tried changing many parameters but still the same. I'm using CDH 5.12.0.

 

Thanks

New Contributor
Posts: 1
Registered: ‎10-04-2017

Re: HDFS mover does not complete in one execution

Hi, 

 

I have the same problem. Did you solve this problem?

Highlighted
Cloudera Employee
Posts: 4
Registered: ‎02-17-2016

Re: HDFS mover does not complete in one execution

Hi Wiktorio,

 

this topic came up when I searched for the same issue, it seems to be a bug that was introduced with a change in CDH5.10.0 and above versions due to HDFS-11015.

 

I am working on a fix under HDFS-13174, is it possible that you experienced the same issue as described there?

 

Istvan

New Contributor
Posts: 2
Registered: ‎08-02-2017

Re: HDFS mover does not complete in one execution

Hi Istvan,

 

This is not the same issue. I didn't receive any error, after some iterations it just stopped.

 

I went through the source code and found out that this is a bug (or a feature).

 

As a workaroud I managed to write a script that executed mover in a loop until data was
migrated completely.

 

Regards,

Wiktor

 

Announcements