Created 08-08-2018 01:48 AM
There is a flow which would generate flowfiles,set input parameters using update attribute and then process it by execute stream command.This would however place the required files in the location as mentioned in update attribute processor.Is there any way which could fetch the processed files directly from the location so as to move the processed files into another location using another processor.
Created 08-08-2018 02:25 AM
You can use ListFile/ListHDFS processor to check for the new files that are created in the directories.
Then use FetchFile(Fetch + move)/MoveHDFS processors to move the fetched files to other location.
Created 08-08-2018 02:30 AM
@Shu thanks for the same,so the flow should be like generate flowfile->update attribute->execute stream->list file->fetch file?.I needed all the processing to happen in one flow and not two separate flows.
Created 08-08-2018 02:43 AM
If you know the filename and path that the file has stored in the directory then you can do this in one flow..
generate flowfile->update attribute->execute stream->fetch file
If you don't know the filename that you have stored using execute Stream command processor then
Need to use as a separate process to list the files in the directory and move them to another location.
Another approach is to after execute stream command processor use another execute stream command processor to list the files and extract the file names as attributes then use Fetch File processor to move another location.
generate flowfile->update attribute->execute stream->execute stream->Split Text -> Extract Text -> fetch file
Refer to this link describes how to get files from the directory.
In addition If your filename having some timestamp value then using route on attribute you can check the filename values also.
Created 08-08-2018 07:06 AM
thanks for the detailed explaination.If the filename keeps changing like "YYYY_MM_DD_HH_MI_SS.csv" ,how can we mention the same in fetch file's filename
Created 08-08-2018 12:20 PM
if you are using
Method1: Then ListFIle processor output's flowfile with filename attribute associated with it.
In Method2: We are listing out the files in the directory then preparing filename attribute to the flowfile. Then Fetch File Processor Fetches the files in the directory.
In both methods works if the filename keep changing also, as we are dynamically adding filename attribute to the flowfile.