Skip to main content
Bonthala_Deepu
Bonthala_Deepu
Silver ⭐️⭐️
August 12, 2022
Question

Unable to import python external modules in nifi execute script

Tried executing following python script body & Module directory as 

import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
#from google.cloud import bigquery
#from google.oauth2 import service_account
import google

class ModJSON(StreamCallback):
    def __init__(self):
        pass
    def process(self, inputStream, outputStream):
        text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
        obj = json.loads(text)
        print(text)
        outputStream.write(bytearray(json.dumps(obj, indent=4).encode('utf-8')))

flowFile = session.get()
if (flowFile != None):
  flowFile = session.write(flowFile, ModJSON())
  flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename')+'_translated.json')
  session.transfer(flowFile, REL_SUCCESS)
session.commit()

 

Module Directory 

/Users/bonthala/Library/Python/2.7/lib/python/site-packages/

 

 

Received the following error

2022-08-13 00:14:56,948 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=8df016ea-0182-1000-35d7-40765b666df2] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines: java.io.IOException: Mark invalid
↳ causes: java.io.IOException: Mark invalid
	at java.base/java.io.BufferedReader.reset(BufferedReader.java:517)
	at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:133)
	at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:320)
	at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:316)
	at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:308)
	at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87)
	at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76)
	at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42)
	at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56)
	at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250)
	at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:211)
	at org.apache.nifi.processors.script.ExecuteScript.setup(ExecuteScript.java:176)
	at jdk.internal.reflect.GeneratedMethodAccessor552.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1653)
	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
java.io.IOException: java.io.IOException: Mark invalid

↳ causes: javax.script.ScriptException: java.io.IOException: java.io.IOException: Mark invalid
↳ causes: org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines
org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines
	at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:256)
	at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:211)
	at org.apache.nifi.processors.script.ExecuteScript.setup(ExecuteScript.java:176)
	at jdk.internal.reflect.GeneratedMethodAccessor552.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
	at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
	at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$6(StandardProcessorNode.java:1653)
	at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.script.ScriptException: java.io.IOException: java.io.IOException: Mark invalid
	at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:226)
	at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:93)
	at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76)
	at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42)
	at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56)
	at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250)
	... 16 common frames omitted
Caused by: org.python.core.PyException: null
	at org.python.core.Py.JavaError(Py.java:552)
	at org.python.core.ParserFacade.fixParseError(ParserFacade.java:107)
	at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:136)
	at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:320)
	at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:316)
	at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:308)
	at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87)
	... 20 common frames omitted
Caused by: java.io.IOException: Mark invalid
	at java.base/java.io.BufferedReader.reset(BufferedReader.java:517)
	at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:133)
	... 24 common frames omitted

 

Python sys path

~ » python                                                                                                                                                              bonthala@Offfice-Admins-MacBook-Pro

WARNING: Python 2.7 is not recommended.
This version is included in macOS for compatibility with legacy software.
Future versions of macOS will not include Python 2.7.
Instead, it is recommended that you transition to using 'python3' from within Terminal.

Python 2.7.16 (default, Jul  5 2020, 02:24:03)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.21) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Users/bonthala/Library/Python/2.7/lib/python/site-packages', '/Library/Python/2.7/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages/virtualenv-20.0.27-py2.7.egg', '/Library/Python/2.7/site-packages/importlib_resources-3.0.0-py2.7.egg', '/Library/Python/2.7/site-packages/filelock-3.0.12-py2.7.egg', '/Library/Python/2.7/site-packages/distlib-0.3.1-py2.7.egg', '/Library/Python/2.7/site-packages/appdirs-1.4.4-py2.7.egg']

 

Can anyone help me to resolve the error and import the modules to the script ?

 

    8 replies

    janis-ax
    janis-ax
    Silver ⭐️⭐️⭐️
    August 22, 2022

    Hi, 
    I'm not 100% sure, but I think you mix some things up. NiFi is written in Java and therefore running in the JVM. In NiFi you can run Jython which is a Python implementation in Java. 

     I guess you are trying to use CPython (mostly the default Python) modules / packages in Jython (Java-Python) which does not work. Maybe you are importing them wrong. 

    Translate
    Bonthala_Deepu
    Bonthala_DeepuAuthor
    Silver ⭐️⭐️
    November 23, 2022

    Hi,

     

    Can you help to run a python script in NIFI containing an external package i.e., import google in this case?

    Translate
    lgallindo
    lgallindo
    Silver ⭐️
    November 22, 2022

    I get the same error with a very simple import:

    try
        import org.apache.commons.io
        direct_nifi = True
    except ImportError:
        direct_nifi = False

    nifi-app.log

    2022-11-22 14:11:52,536 INFO [pool-7-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
    2022-11-22 14:11:52,537 INFO [pool-7-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 0 milliseconds
    2022-11-22 14:11:55,817 INFO [NiFi Web Server-137] o.a.n.c.s.StandardProcessScheduler Running once ExecuteScript[id=a00fd11d-0184-1000-fa55-6eabbd1be8fe]
    2022-11-22 14:11:55,817 INFO [NiFi Web Server-137] o.a.n.controller.StandardProcessorNode Starting ExecuteScript[id=a00fd11d-0184-1000-fa55-6eabbd1be8fe]
    2022-11-22 14:11:56,119 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@6eea35ff // Another save pending = false
    2022-11-22 14:11:56,765 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=a00fd11d-0184-1000-fa55-6eabbd1be8fe] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines
    org.apache.nifi.processor.exception.ProcessException: Could not instantiate script engines
    at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:256)
    at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:211)
    at org.apache.nifi.processors.script.ExecuteScript.setup(ExecuteScript.java:178)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
    at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
    at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
    at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
    at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1733)
    at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
    Caused by: javax.script.ScriptException: SyntaxError: mismatched input '\r\n' expecting COLON in <script> at line number 23 at column number 3
    at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:213)
    at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:93)
    at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76)
    at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42)
    at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56)
    at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250)
    ... 17 common frames omitted
    Caused by: org.python.core.PySyntaxError: SyntaxError: mismatched input '\r\n' expecting COLON (<script>, line 23)
    at org.python.core.ParserFacade.fixParseError(ParserFacade.java:95)
    at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:136)
    at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:321)
    at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:317)
    at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:309)
    at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87)
    ... 21 common frames omitted

    Translate
    Bonthala_Deepu
    Bonthala_DeepuAuthor
    Silver ⭐️⭐️
    November 23, 2022

    Hi @lgallindo ,

     

    Seems this is a syntax error. See the following error log 

     

    Caused by: javax.script.ScriptException: SyntaxError: mismatched input '\r\n' expecting COLON in <script> at line number 23 at column number 3
    at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:213)
    at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:93)
    at org.python.jsr223.PyScriptEngine.compile(PyScriptEngine.java:76)
    at org.apache.nifi.script.impl.JythonScriptRunner.<init>(JythonScriptRunner.java:42)
    at org.apache.nifi.script.ScriptRunnerFactory.createScriptRunner(ScriptRunnerFactory.java:56)
    at org.apache.nifi.script.ScriptingComponentHelper.setupScriptRunners(ScriptingComponentHelper.java:250)
    ... 17 common frames omitted
    Caused by: org.python.core.PySyntaxError: SyntaxError: mismatched input '\r\n' expecting COLON (<script>, line 23)
    at org.python.core.ParserFacade.fixParseError(ParserFacade.java:95)
    at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:136)
    at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:321)
    at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:317)
    at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:309)
    at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:87)
    ... 21 common frames omitted

    Translate
    joseomjr
    joseomjr
    Gold ⭐️⭐️
    June 14, 2023

    Was getting the same error with some of our Python/Jython scripts and how I got it to work was remove the modules path from "Module Directory" and add it with the following at the very top of your script.

     

    import sys
    sys.path.append("/Users/bonthala/Library/Python/2.7/lib/python/site-packages/")
    Translate
      Kiranq
      Kiranq
      Silver ⭐️⭐️⭐️
      August 28, 2023

      @joseomjr I am trying to replicate your suggested method, please answer did you include the import command after sys.path.append command? 

      Translate
      joseomjr
      joseomjr
      Gold ⭐️⭐️
      August 28, 2023

      The 2 lines I provided are the very first lines and then all your code comes after. 

      Translate
      Kiranq
      Kiranq
      Silver ⭐️⭐️⭐️
      August 28, 2023

      Thanks for the reply.
      I did it in the same way you have mentioned (I am importing Selenium module and from this module, Service, WebDriverWait and EC components are imported) but late in the code execution, my scripts gives an error "global name Service is not defined".


      Any idea whats causing the issue?

      Translate
      Terms & ConditionsAccessibility statement