question Re: trying to get the most basic python UDFs working in Support Questions

trying to get the most basic python UDFs working

mew — Tue, 29 Oct 2024 17:10:19 GMT

pig-0.17.0bin/pig -x local

very basic UDF file:

#!/usr/bin/python3

from pig_util import outputSchema

@outputSchema("as:int")
def square(num):
if num == None:
return None
return ((num) * (num))

@outputSchema("word:chararray")
def concat(word):
return word + word

Exceedingly simple pig script:

REGISTER '/home/scs/woodcock/SD411/lab_udf/test.py' USING org.apache.pig.scripting.streaming.Python.PythonScriptEngine AS myFuncs;

A = LOAD '/home/scs/woodcock/SD411/DATA/accident.csv' USING PigStorage(',') AS (state:int,name:chararray);

B = FOREACH A GENERATE myFuncs.square(state) AS state, name;

If I do a "DUMP A" I get exactly what I would expect.

But, on a "DUMP B", I get a failed job:

java.lang.Exception: org.apache.pig.impl.streaming.StreamingUDFException: LINE :
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: org.apache.pig.impl.streaming.StreamingUDFException: LINE :
at org.apache.pig.impl.builtin.StreamingUDF$ProcessErrorThread.run(StreamingUDF.java:506)

grunt> Exception in thread "Thread-82" java.lang.NullPointerException: Cannot invoke "java.util.concurrent.BlockingQueue.put(Object)" because the return value of "org.apache.pig.impl.builtin.StreamingUDF.access$500(org.apache.pig.impl.builtin.StreamingUDF)" is null
at org.apache.pig.impl.builtin.StreamingUDF$ProcessOutputThread.run(StreamingUDF.java:471)
2024-10-29 13:02:15,296 [communication thread] INFO org.apache.hadoop.mapred.LocalJobRunner - map > map

Re: trying to get the most basic python UDFs working

DianaTorres — Tue, 29 Oct 2024 18:40:31 GMT

@mew Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our MapReduce experts @Stella Tang @vchalla @jeniferA who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: trying to get the most basic python UDFs working

mew — Thu, 31 Oct 2024 15:46:45 GMT

Honestly, at this point, I would probably accept any (less trivial than a "HelloWorld"--that is something that actually computes, not just returns a fixed string) Python UDF and the script that will work in pig17. I feel like I'm just cutting and pasting the standard documented examples, and that's not close to working, which isn't giving me a great feeling.

Re: trying to get the most basic python UDFs working

mew — Fri, 01 Nov 2024 17:05:18 GMT

I also note that I can get Java UDFs to work; so its not a general UDF problem...it's something specific to Python.

Re: trying to get the most basic python UDFs working

mew — Mon, 04 Nov 2024 14:06:46 GMT

Hmmm...so I tried rolling back to Pig 13, and somewhat troubling...but that totally worked. On multiple different machines. Perhaps something didn't get tested real well before release?

Re: trying to get the most basic python UDFs working

mew — Tue, 05 Nov 2024 20:09:42 GMT

Here is the actual fix (it's actually quite loony): don't wrap the name of the file in the REGISTER statement with single quotes. That's it.

Catastrophic problems here:

1) Obviously not backwards compatible.

2) If this is a problem, why not just indicate that a) the format is wrong, or b) that a path that started with a single-quote did not yield a valid python file or c) anything understandable instead of getting in the middle of the M/R computation and throwing wacky (mkey? nullPointer) errors.