processor to output the files from execute stream command

[#1]

There is a flow which would generate flowfiles,set input parameters using update attribute and then process it by execute stream command.This would however place the required files in the location as mentioned in update attribute processor.Is there any way which could fetch the processed files directly from the location so as to move the processed files into another location using another processor.

[#2]

@Gillu Varghese

You can use ListFile/ListHDFS processor to check for the new files that are created in the directories.

Then use FetchFile(Fetch + move)/MoveHDFS processors to move the fetched files to other location.

[#3]

@Shu thanks for the same,so the flow should be like generate flowfile->update attribute->execute stream->list file->fetch file?.I needed all the processing to happen in one flow and not two separate flows.

[#4]

@Gillu Varghese

If you know the filename and path that the file has stored in the directory then you can do this in one flow..

flow:

generate flowfile->update attribute->execute stream->fetch file

If you don't know the filename that you have stored using execute Stream command processor then
Method1:

Need to use as a separate process to list the files in the directory and move them to another location.

Method2:

Another approach is to after execute stream command processor use another execute stream command processor to list the files and extract the file names as attributes then use Fetch File processor to move another location.

Flow:

generate flowfile->update attribute->execute stream->execute stream->Split Text -> Extract Text -> fetch file

Refer to this link describes how to get files from the directory.

In addition If your filename having some timestamp value then using route on attribute you can check the filename values also.

[#5]

@Shu

thanks for the detailed explaination.If the filename keeps changing like "YYYY_MM_DD_HH_MI_SS.csv" ,how can we mention the same in fetch file's filename

[#6]

@Gillu Varghese

if you are using

Method1: Then ListFIle processor output's flowfile with filename attribute associated with it.

In Method2: We are listing out the files in the directory then preparing filename attribute to the flowfile. Then Fetch File Processor Fetches the files in the directory.

In both methods works if the filename keep changing also, as we are dynamically adding filename attribute to the flowfile.