Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

NiFi - Execute python script

I have a NiFi cluster (1.4.0) . I have already created a Nifi process flow with ListFile, FetchFile PutHDFS processors to read files and put it into HDFS

I need suggestions/ideas to implement generic process flow in Nifi for the below use case.

1. Read property file from specified location (vendor name|trans type|file pattern) for every hour.

2. Send Process Start Notification for Each property Entry

3. Count number of files for the specified file pattern in the directory.

4. Pass each property entry values as an arguments to the already created Nifi Process flow.

5. Send Process End Notification for Each property Entry with the audit details.

For Step#1 I am planning to use Python Script to read file and set Property values for the already created process.

Please let me know how to configure python script in Nifi or is there any Processor available to do the same in Nifi.

6 REPLIES 6

Super Guru

Can you explain more about what you're trying to do in Step 1? Examples locations, filenames, etc. This sounds like something you should be able to do without a scripting processor.

@Matt Burgess

In Step#1. I am planning to Read Property File from specified location on server in Python and set Property values for each entry and invoke my current processor.

vendorName=XYZ

HDFSLocation= hdfs:///vendorName/<current_date>/

FilePattern=*.CSV or TSV or Zip

Please share your thoughts on the implementation.

Super Guru

So the goal is to read in a property file, extract the properties to flow file attributes, and send the flow file downstream to Step 2?

@Matt Burgess

Yes.Kindly post your ideas which is best-fit for my requirement.

Super Guru

AFAIK there is no existing processor, but it can be done quickly and easily in Groovy or Javascript using ExecuteScript. You can use Jython if you wish, but Jython is slower than the other two I mentioned.

Master Guru

@BRAMHA NAIDU AELEM

Suggestion: Using the @<username> (as i have above) in your responses will trigger a notification to that user. Otherwise user may not know you have responded to them. As this is an open forum, user may not be actively logging in and searching through all open questions.