There is a flow which would generate flowfiles,set input parameters using update attribute and then process it by execute stream command.This would however place the required files in the location as mentioned in update attribute processor.Is there any way which could fetch the processed files directly from the location so as to move the processed files into another location using another processor.
You can use ListFile/ListHDFS processor to check for the new files that are created in the directories.
Then use FetchFile(Fetch + move)/MoveHDFS processors to move the fetched files to other location.
@Shu thanks for the same,so the flow should be like generate flowfile->update attribute->execute stream->list file->fetch file?.I needed all the processing to happen in one flow and not two separate flows.
If you know the filename and path that the file has stored in the directory then you can do this in one flow..
generate flowfile->update attribute->execute stream->fetch file
If you don't know the filename that you have stored using execute Stream command processor then
Need to use as a separate process to list the files in the directory and move them to another location.
Another approach is to after execute stream command processor use another execute stream command processor to list the files and extract the file names as attributes then use Fetch File processor to move another location.
generate flowfile->update attribute->execute stream->execute stream->Split Text -> Extract Text -> fetch file
Refer to this link describes how to get files from the directory.
In addition If your filename having some timestamp value then using route on attribute you can check the filename values also.
if you are using
Method1: Then ListFIle processor output's flowfile with filename attribute associated with it.
In Method2: We are listing out the files in the directory then preparing filename attribute to the flowfile. Then Fetch File Processor Fetches the files in the directory.
In both methods works if the filename keep changing also, as we are dynamically adding filename attribute to the flowfile.