Hi, Since we can mention only one prefix in ListS3 processor I am trying to access AWS S3 using Python boto3 in NiFi ExecuteScript processor. If this succeeds, I can send a list of folder paths to the python script to get files from various folders under S3 bucket. I provided the path of boto3-1.6.0.tar.gz in ModuleDirectory but I get the below error.
ExecuteScript Processor :
ExecuteScript[id=d6cf51e8-0161-1000-32de-7748af781842] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: ImportError: No module named boto3 in <script> at line number 1: javax.script.ScriptException: ImportError: No module named boto3 in <script> at line number 1
Set your module direct to something like this to pick up all the python modules, /usr/local/lib/python2.7/site-packages,/usr/lib/python2.7/dist-packages
As NiFi is a pure Java/JVM application, we use Jython rather than Python for ExecuteScript. This means that you can't import native (CPython, e.g.) modules into your Jython scripts in ExecuteScript
consider using ExecuteStreamCommand with a real Python interpreter and script. I'm looking at Py4J to try and bridge the gap