question Re: Pig: Streaming through python in Support Questions

Pig: Streaming through python

Eukrev — Wed, 20 Apr 2016 09:26:13 GMT

Hi,

I have a small Voters list(name,gender,place,age) where I wanted to eliminate the voters whose age is <= 20. I wanted to try streaming in pig.

When I run the dump on stream its fails and is unable to idenetify python commands. I have attached python script, input data file, pig script and log file. Could you guide where should I install the python in Sandbox. Thank you.

Input:

AAA,Female,Blr,40 
BBB,Female,London,35
YYY,Female,Pondy,12
JJJ,Male,London,4
SSS,Female,Pondy,30

pig script in tez_local mode:

grunt> Voters = LOAD 'file:///user/revathy/pig/Voters.txt' USING PigStorage(',') AS (VoterName:chararray,Gender:chararray,Place:chararray,Age:int); 
 
grunt> Eligible = STREAM Voters THROUGH `/root/revathy/pig/hello.py` AS (VoterName:chararray,Gender:chararray,Place:chararray,Age:int);

Python script:(Tested in Python editor)

import sys 
THRESHOLD = 20 def filterVal(line,val4): 
if int(val4) > THRESHOLD: 
          sys.stdout.writelines(line) 
         
        return 
try: 
         
    for line in sys.stdin.readlines(): 
                 val1,val2,val3,val4 = str(line).split(",") 
                 filterVal(line,val4) 
except: 
         
print "Error in try block"

Log:

/root/revathy/pig/hello.py: 
line 1: import: command not found                                                                                        
/root/revathy/pig/hello.py:
line 2: THRESHOLD: command not found                                                                                    
/root/revathy/pig/hello.py: 
line 3:                                                                                                                  
: command not found

Re: Pig: Streaming through python

aervits — Wed, 20 Apr 2016 17:18:34 GMT

You did not include the python interpreter line in your python script and it has difficulty understanding its python. For what you're trying to achieve, you can skip streaming and just use Pig built-in filter function. It will perform better than streaming. http://pig.apache.org/docs/r0.15.0/

SSN_NAME = load 'students.txt' using PigStorage() as (ssn:long, name:chararray);

/* do a left outer join of SSN with SSN_Name */
X = JOIN SSN by ssn LEFT OUTER, SSN_NAME by ssn;

/* only keep those ssn's for which there is no name */
Y = filter X by IsEmpty(SSN_NAME);

Re: Pig: Streaming through python

Eukrev — Thu, 21 Apr 2016 06:17:45 GMT

Thank you for providing an alternation approach. I am learning Pig and would like to try the stream command - see how to run python in pig.

Is this the line, to be added as first line so that execution engine understands its python? #! /usr/bin/env python I tried but still get the same error. Could you please help. Thank you!!!

Re: Pig: Streaming through python

aervits — Thu, 21 Apr 2016 07:04:37 GMT

Checkout my UDF examples using streaming https://github.com/dbist/pig/tree/master/udfs

specifically formathtml.pig script and it's associated UDF written in python

Re: Pig: Streaming through python

Eukrev — Thu, 21 Apr 2016 23:31:48 GMT

Thank you. Its a good simple example for me to understand.

Re: Pig: Streaming through python

betocolsf — Tue, 11 Jun 2019 00:59:48 GMT

Do you know if there is a way to specify a python virtual environment for streaming_python to use instead of it using the base python installation?