Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - Execute python script

NiFi - Execute python script

New Contributor

I have a NiFi cluster (1.4.0) . I have already created a Nifi process flow with ListFile, FetchFile PutHDFS processors to read files and put it into HDFS

I need suggestions/ideas to implement generic process flow in Nifi for the below use case.

1. Read property file from specified location (vendor name|trans type|file pattern) for every hour.

2. Send Process Start Notification for Each property Entry

3. Count number of files for the specified file pattern in the directory.

4. Pass each property entry values as an arguments to the already created Nifi Process flow.

5. Send Process End Notification for Each property Entry with the audit details.

For Step#1 I am planning to use Python Script to read file and set Property values for the already created process.

Please let me know how to configure python script in Nifi or is there any Processor available to do the same in Nifi.

6 REPLIES 6
Highlighted

Re: NiFi - Execute python script

Super Guru

Can you explain more about what you're trying to do in Step 1? Examples locations, filenames, etc. This sounds like something you should be able to do without a scripting processor.

Highlighted

Re: NiFi - Execute python script

New Contributor

@Matt Burgess

In Step#1. I am planning to Read Property File from specified location on server in Python and set Property values for each entry and invoke my current processor.

vendorName=XYZ

HDFSLocation= hdfs:///vendorName/<current_date>/

FilePattern=*.CSV or TSV or Zip

Please share your thoughts on the implementation.

Highlighted

Re: NiFi - Execute python script

Super Guru

So the goal is to read in a property file, extract the properties to flow file attributes, and send the flow file downstream to Step 2?

Highlighted

Re: NiFi - Execute python script

New Contributor

@Matt Burgess

Yes.Kindly post your ideas which is best-fit for my requirement.

Highlighted

Re: NiFi - Execute python script

Super Guru

AFAIK there is no existing processor, but it can be done quickly and easily in Groovy or Javascript using ExecuteScript. You can use Jython if you wish, but Jython is slower than the other two I mentioned.

Highlighted

Re: NiFi - Execute python script

Master Guru

@BRAMHA NAIDU AELEM

Suggestion: Using the @<username> (as i have above) in your responses will trigger a notification to that user. Otherwise user may not know you have responded to them. As this is an open forum, user may not be actively logging in and searching through all open questions.

Don't have an account?
Coming from Hortonworks? Activate your account here