Quoting directly from the official docu: FetchSMB: "Fetches files from a SMB Share. Designed to be used in tandem with ListSmb." My workflow is pretty simple, I have ListSMB which reads from a shared network directory, connected to FetchSMB which should get 1 file (the processor forces you to put the path of a specific file or it won't work) and finally PutHDFS to write all the files in a distributed file system (hadoop). I don't / can't understand why whenever I use List and Fetch in tandem the pipeline will just take all the files it can find on the directory specified in the List processor but won't write to the HDFS the single file I specified on the Fetch, instead it will write on the HDFS all the files from the List. The fetch, directly connected to the HDFS, should pass only a single file. What am I missing here ? What is the purpose of the fetch if it is "by-passed" by the list processor ? On top of that, all the file written to the HDFS will pick the same weight (KB or MB) of the single file I specified on the Fetch, resulting in many files being broken / corrupted. As you can see from the pic above. They SHOULD NOT have the same weight. I thought that list + fetch were used primarily to move big data, but then I ask myself why the Fetch processor wants me to indicate the path of a specific file and then ignore it ? I didn't find the documentation helpful at all in this regard. Thank you and have a nice day. EDIT: Added some screenshots for clarity. Dataflow (pretty simple): ListSMB: (I am using "no tracking" just to make testing easier and faster. In prod we will have to set up a tracking strategy.) FetchSMB: PutHDFS: This is a sample screenshot of the queue list (between the list and the fetch processors): This is another screenshot of the queue (between fetch and putHDFS processors) As you can see the "file size" is over 60mb, but it's just a .png which should be no more than 300kb. 60mb is the file size of the file specified in the properties of the Fetch processor (20211021_PL_GIO.zip) I hope that it's better now! If you need other info or screenshots just let me know. Have a nice day.
... View more
