question Re: cannot use numpy or scipy in python in nifi executescript in Archives of Support Questions (Read Only)

cannot use numpy or scipy in python in nifi executescript

121285904 — Sat, 27 Aug 2016 21:15:18 GMT

I used the python in executescript.

I installed numpy/ scipy, but always got the error: 'ImportError: No module named type_check'

I run the same code in jupyter successfully.

Any suggestion? Thanks in advance.

Re: cannot use numpy or scipy in python in nifi executescript

mburgess — Sat, 27 Aug 2016 21:40:29 GMT

Although the script engine reports its name as "python", it is actually Jython, which can only use pure Python modules, not native modules like numpy/scipy. If this is needed, consider ExecuteProcess or (if you have incoming flow files) ExecuteStreamCommand which can execute the command-line python.

Re: cannot use numpy or scipy in python in nifi executescript

121285904 — Sun, 28 Aug 2016 19:18:12 GMT

Thanks for your explain.


	I used ExecuteScript as below:


from scipy.stats import f_oneway
import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback




class PyStreamCallback(StreamCallback):
  def __init__(self):
        pass
  def process(self, inputStream, outputStream):
    jsonData = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
   
    data = json.loads(jsonData)


    values = [i['fltValue'] for i in data["data"]]
    firsts = [i['first'] for i in data["data"]]
    seconds = [i['second'] for i in data["data"]]


    f,p = f_oneway(values,firsts,seconds)
    data["f"] = f
    data["p"] = p
              
    outputStream.write(bytearray(json.dumps(newObj, indent=4).encode('utf-8'))) 


flowFile = session.get()
if (flowFile != None):
  flowFile = session.write(flowFile,PyStreamCallback())
  session.transfer(flowFile, REL_SUCCESS)


how to use these 2 processor instead of?

Re: cannot use numpy or scipy in python in nifi executescript

vishnu-priya_ku — Mon, 21 Nov 2016 20:15:03 GMT

So what about importing modules like paramiko, pysftp?

Re: cannot use numpy or scipy in python in nifi executescript

mburgess — Tue, 22 Nov 2016 02:38:53 GMT

Paramiko uses Crypto which is a native module, so this is not pure Python either and cannot be used in ExecuteScript. ExecuteProcess or ExecuteStreamCommand should work though.

Re: cannot use numpy or scipy in python in nifi executescript

TimothySpann — Sat, 11 Mar 2017 05:30:41 GMT

seems nltk has native stuff too.

That should be explicit

Re: cannot use numpy or scipy in python in nifi executescript

vincentvanouden — Thu, 15 Mar 2018 00:31:38 GMT

Hi Boyer, did you end up getting this to work? Did you use ExecuteStreamCommand? If so, would you mind sharing the configuration of your processor (as in how is the Python script called), and lastly, how do you ingest the FlowFile in your python script?

Any advice is much appreciated.

Re: cannot use numpy or scipy in python in nifi executescript

TimothySpann — Thu, 15 Mar 2018 00:58:49 GMT

You have to make sure you install the libraries on the server for the correct python version.

I always run with executeprocess or executestreamcommand and wrap my python in a shell

Re: cannot use numpy or scipy in python in nifi executescript

kirilm — Tue, 12 Jun 2018 17:53:38 GMT

hi Timothy,

can you explain this a bit?

is it possible to execute a script.py with #!/usr/bin/python inside and then run it as a bash with NiFi?

Or how ist this possible to wrap python in a shell? would help me a lot to start my python scripts with sklearn / numpy / pandas inside NiFi, then grab the exports.csv or exports.json and go ahead with my NiFi worlkflow for FlowFiles.

cheers