I have a NiFi cluster (1.4.0) . I have already created a Nifi process flow with ListFile, FetchFile PutHDFS processors to read files and put it into HDFS
I need suggestions/ideas to implement generic process flow in Nifi for the below use case.
1. Read property file from specified location (vendor name|trans type|file pattern) for every hour.
2. Send Process Start Notification for Each property Entry
3. Count number of files for the specified file pattern in the directory.
4. Pass each property entry values as an arguments to the already created Nifi Process flow.
5. Send Process End Notification for Each property Entry with the audit details.
For Step#1 I am planning to use Python Script to read file and set Property values for the already created process.
Please let me know how to configure python script in Nifi or is there any Processor available to do the same in Nifi.
Can you explain more about what you're trying to do in Step 1? Examples locations, filenames, etc. This sounds like something you should be able to do without a scripting processor.
In Step#1. I am planning to Read Property File from specified location on server in Python and set Property values for each entry and invoke my current processor.
FilePattern=*.CSV or TSV or Zip
Please share your thoughts on the implementation.
So the goal is to read in a property file, extract the properties to flow file attributes, and send the flow file downstream to Step 2?
Suggestion: Using the @<username> (as i have above) in your responses will trigger a notification to that user. Otherwise user may not know you have responded to them. As this is an open forum, user may not be actively logging in and searching through all open questions.