question Unable to import python external modules in nifi execute script in Support Questions

Unable to import python external modules in nifi execute script

Bonthala_Deepu — Fri, 12 Aug 2022 19:31:33 GMT

Tried executing following python script body & Module directory as

import json import java.io from org.apache.commons.io import IOUtils from java.nio.charset import StandardCharsets from org.apache.nifi.processor.io import StreamCallback #from google.cloud import bigquery #from google.oauth2 import service_account import google class ModJSON(StreamCallback): def __init__(self): pass def process(self, inputStream, outputStream): text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) obj = json.loads(text) print(text) outputStream.write(bytearray(json.dumps(obj, indent=4).encode('utf-8'))) flowFile = session.get() if (flowFile != None): flowFile = session.write(flowFile, ModJSON()) flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename')+'_translated.json') session.transfer(flowFile, REL_SUCCESS) session.commit()

Module Directory

/Users/bonthala/Library/Python/2.7/lib/python/site-packages/

Received the following error

2022-08-13 00:14:56,948 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=8df016ea-0182-1000-35d7-40765b666df2] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines: java.io.IOException: Mark invalid ↳ causes: java.io.IOException: Mark invalid at java.base/java.io.BufferedReader.reset(BufferedReader.java:517) at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:133) at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:320) at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:316) at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:308) at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87) at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76) at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42) at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56) at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250) at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:211) at org.apache.nifi.processors.script.ExecuteScript.setup(ExecuteScript.java:176) at jdk.internal.reflect.GeneratedMethodAccessor552.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55) at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1653) at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) java.io.IOException: java.io.IOException: Mark invalid ↳ causes: javax.script.ScriptException: java.io.IOException: java.io.IOException: Mark invalid ↳ causes: org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:256) at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:211) at org.apache.nifi.processors.script.ExecuteScript.setup(ExecuteScript.java:176) at jdk.internal.reflect.GeneratedMethodAccessor552.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55) at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1653) at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: javax.script.ScriptException: java.io.IOException: java.io.IOException: Mark invalid at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:226) at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:93) at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76) at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42) at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56) at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250) ... 16 common frames omitted Caused by: org.python.core.PyException: null at org.python.core.Py.JavaError(Py.java:552) at org.python.core.ParserFacade.fixParseError(ParserFacade.java:107) at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:136) at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:320) at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:316) at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:308) at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87) ... 20 common frames omitted Caused by: java.io.IOException: Mark invalid at java.base/java.io.BufferedReader.reset(BufferedReader.java:517) at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:133) ... 24 common frames omitted

Python sys path

~ » python bonthala@Offfice-Admins-MacBook-Pro WARNING: Python 2.7 is not recommended. This version is included in macOS for compatibility with legacy software. Future versions of macOS will not include Python 2.7. Instead, it is recommended that you transition to using 'python3' from within Terminal. Python 2.7.16 (default, Jul 5 2020, 02:24:03) [GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.21) (-macos10.15-objc- on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Users/bonthala/Library/Python/2.7/lib/python/site-packages', '/Library/Python/2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages/virtualenv-20.0.27-py2.7.egg', '/Library/Python/2.7/site-packages/importlib_resources-3.0.0-py2.7.egg', '/Library/Python/2.7/site-packages/filelock-3.0.12-py2.7.egg', '/Library/Python/2.7/site-packages/distlib-0.3.1-py2.7.egg', '/Library/Python/2.7/site-packages/appdirs-1.4.4-py2.7.egg']

Can anyone help me to resolve the error and import the modules to the script ?

Re: Unable to import python external modules in nifi execute script

janis-ax — Mon, 22 Aug 2022 15:14:58 GMT

Hi,
I'm not 100% sure, but I think you mix some things up. NiFi is written in Java and therefore running in the JVM. In NiFi you can run Jython which is a Python implementation in Java.

I guess you are trying to use CPython (mostly the default Python) modules / packages in Jython (Java-Python) which does not work. Maybe you are importing them wrong.

Re: Unable to import python external modules in nifi execute script

lgallindo — Tue, 22 Nov 2022 17:16:42 GMT

I get the same error with a very simple import:

try
    import org.apache.commons.io
    direct_nifi = True
except ImportError:
    direct_nifi = False

nifi-app.log

2022-11-22 14:11:52,536 INFO [pool-7-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2022-11-22 14:11:52,537 INFO [pool-7-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
2022-11-22 14:11:55,817 INFO [NiFi Web Server-137] o.a.n.c.s.StandardProcessScheduler Running once ExecuteScript[id=a00fd11d-0184-1000-fa55-6eabbd1be8fe]
2022-11-22 14:11:55,817 INFO [NiFi Web Server-137] o.a.n.controller.StandardProcessorNode Starting ExecuteScript[id=a00fd11d-0184-1000-fa55-6eabbd1be8fe]
2022-11-22 14:11:56,119 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@6eea35ff // Another save pending = false
2022-11-22 14:11:56,765 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=a00fd11d-0184-1000-fa55-6eabbd1be8fe] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines
org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines
at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:256)
at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:211)
at org.apache.nifi.processors.script.ExecuteScript.setup(ExecuteScript.java:178)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1733)
at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.script.ScriptException: SyntaxError: mismatched input '\r\n' expecting COLON in <script> at line number 23 at column number 3
at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:213)
at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:93)
at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76)
at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42)
at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56)
at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250)
... 17 common frames omitted
Caused by: org.python.core.PySyntaxError: SyntaxError: mismatched input '\r\n' expecting COLON (<script>, line 23)
at org.python.core.ParserFacade.fixParseError(ParserFacade.java:95)
at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:136)
at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:321)
at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:317)
at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:309)
at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87)
... 21 common frames omitted

Re: Unable to import python external modules in nifi execute script

Bonthala_Deepu — Wed, 23 Nov 2022 09:26:40 GMT

Hi,

Can you help to run a python script in NIFI containing an external package i.e., import google in this case?

Re: Unable to import python external modules in nifi execute script

Bonthala_Deepu — Wed, 23 Nov 2022 09:29:02 GMT

Hi @lgallindo ,

Seems this is a syntax error. See the following error log

Caused by: javax.script.ScriptException: SyntaxError: mismatched input '\r\n' expecting COLON in <script> at line number 23 at column number 3
at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:213)
at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:93)
at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76)
at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42)
at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56)
at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250)
... 17 common frames omitted
Caused by: org.python.core.PySyntaxError: SyntaxError: mismatched input '\r\n' expecting COLON (<script>, line 23)
at org.python.core.ParserFacade.fixParseError(ParserFacade.java:95)
at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:136)
at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:321)
at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:317)
at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:309)
at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87)
... 21 common frames omitted

Re: Unable to import python external modules in nifi execute script

joseomjr — Wed, 14 Jun 2023 19:05:39 GMT

Was getting the same error with some of our Python/Jython scripts and how I got it to work was remove the modules path from "Module Directory" and add it with the following at the very top of your script.

import sys

sys.path.append("/Users/bonthala/Library/Python/2.7/lib/python/site-packages/")

Re: Unable to import python external modules in nifi execute script

Kiranq — Mon, 28 Aug 2023 14:07:02 GMT

@joseomjr I am trying to replicate your suggested method, please answer did you include the import command after sys.path.append command?

Re: Unable to import python external modules in nifi execute script

joseomjr — Mon, 28 Aug 2023 14:10:21 GMT

The 2 lines I provided are the very first lines and then all your code comes after.

Re: Unable to import python external modules in nifi execute script

Kiranq — Mon, 28 Aug 2023 14:50:02 GMT

Thanks for the reply.
I did it in the same way you have mentioned (I am importing Selenium module and from this module, Service, WebDriverWait and EC components are imported) but late in the code execution, my scripts gives an error "global name Service is not defined".

Any idea whats causing the issue?