question Re: python script to fetch files in NiFi in Support Questions

python script to fetch files in NiFi

sayak17 — Fri, 01 Jul 2022 08:00:16 GMT

I am learning the ExecuteScript processor. I want to understand, how to use python script to fetch a file from local, similar to what the GetFile processor does.

Thanks

Re: python script to fetch files in NiFi

Brenigan — Fri, 01 Jul 2022 09:55:44 GMT

I think you should refer to Ni-Fi cookbook
That is pretty much THE only instruction for making scripts for ni fi

Re: python script to fetch files in NiFi

SAMSAL — Fri, 01 Jul 2022 12:50:19 GMT

hi,

Is there a reason that you want to create your own fetch files processor vs using the existing processors like: GetFile or FetchFile

Re: python script to fetch files in NiFi

sayak17 — Fri, 01 Jul 2022 13:25:48 GMT

As part of a POC, we want to explore the capabilities of a custom script to fetch files from a source, that pre-existing processors cannot. As a starting step, we are trying to try out writing a script that can fetch file similar to GetFile, to get some basic idea of how that would work. Then we will try out more complicated sources, of which nifi perhaps doesn't provide support by default.

Re: python script to fetch files in NiFi

SAMSAL — Fri, 01 Jul 2022 13:43:45 GMT

If you want you can take a a look at the actual source code for existing procssors like GetFile on Github to see if that will help:

https://github.com/coco11563/nifi-1.4.0/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GetFile.java

Re: python script to fetch files in NiFi

sayak17 — Mon, 04 Jul 2022 14:02:06 GMT

Thanks. Using examples from that cookbook, I could make it work.

Here is my code which works:

from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import OutputStreamCallback class PyOutputStreamCallback(OutputStreamCallback): def __init__(self): pass def process(self, outputStream): with open("D:\\Work\\nifi test\\custom processor input\\random_json.json") as f: file_content = f.read() outputStream.write(bytearray(file_content.encode('utf-8'))) flowFile = session.create() if(flowFile != None): flowFile = session.write(flowFile, PyOutputStreamCallback()) flowFile = session.putAttribute(flowFile, "filename", 'input_file.json') session.transfer(flowFile, REL_SUCCESS) session.commit()

Next, I will figure out if I can add the local directory path as a property and read that, instead of hardcoding it in the script.