Support Questions

Find answers, ask questions, and share your expertise

PutHDFS Isilon Errors

avatar
Explorer

Dear Experts,
We have some files in our HDFS server and want to get these files and put them into Isilon. We use GetHDFS and PutHDFS for this purpose. But, we get the lease and lock errors, as shown in the picture below. How can we fix these problems?
Regardsflow.PNGnifi error.png

1 ACCEPTED SOLUTION

avatar
Super Guru

@Arash ,

 

It's likely that your flow is running on multiple nodes and each node is trying to process every file. When different nodes try to write the same file at the same time you get these errors.

 

You can change your flow to use a List-Fetch pattern. The ListHDFS processor should run only on the primary node and the output of it (the list of files) can be load-balanced across the nodes so that each node will FetchHDFS different files.

 

Cheers,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.

View solution in original post

1 REPLY 1

avatar
Super Guru

@Arash ,

 

It's likely that your flow is running on multiple nodes and each node is trying to process every file. When different nodes try to write the same file at the same time you get these errors.

 

You can change your flow to use a List-Fetch pattern. The ListHDFS processor should run only on the primary node and the output of it (the list of files) can be load-balanced across the nodes so that each node will FetchHDFS different files.

 

Cheers,

André

 

--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs up button.