Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Pig: Streaming through python

avatar
Rising Star

Hi,

I have a small Voters list(name,gender,place,age) where I wanted to eliminate the voters whose age is <= 20. I wanted to try streaming in pig.

When I run the dump on stream its fails and is unable to idenetify python commands. I have attached python script, input data file, pig script and log file. Could you guide where should I install the python in Sandbox. Thank you.

Input:

AAA,Female,Blr,40 
BBB,Female,London,35
YYY,Female,Pondy,12
JJJ,Male,London,4
SSS,Female,Pondy,30

pig script in tez_local mode:

grunt> Voters = LOAD 'file:///user/revathy/pig/Voters.txt' USING PigStorage(',') AS (VoterName:chararray,Gender:chararray,Place:chararray,Age:int); 
 
grunt> Eligible = STREAM Voters THROUGH `/root/revathy/pig/hello.py` AS (VoterName:chararray,Gender:chararray,Place:chararray,Age:int);

Python script:(Tested in Python editor)

import sys 
THRESHOLD = 20 def filterVal(line,val4): 
if int(val4) > THRESHOLD: 
          sys.stdout.writelines(line) 
         
        return 
try: 
         
    for line in sys.stdin.readlines(): 
                 val1,val2,val3,val4 = str(line).split(",") 
                 filterVal(line,val4) 
except: 
         
print "Error in try block"

Log:

/root/revathy/pig/hello.py: 
line 1: import: command not found                                                                                        
/root/revathy/pig/hello.py:
line 2: THRESHOLD: command not found                                                                                    
/root/revathy/pig/hello.py: 
line 3:                                                                                                                  
: command not found              
1 ACCEPTED SOLUTION

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
Master Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Rising Star

Thank you for providing an alternation approach. I am learning Pig and would like to try the stream command - see how to run python in pig.

Is this the line, to be added as first line so that execution engine understands its python? #! /usr/bin/env python I tried but still get the same error. Could you please help. Thank you!!!

avatar
Master Mentor

Checkout my UDF examples using streaming https://github.com/dbist/pig/tree/master/udfs

specifically formathtml.pig script and it's associated UDF written in python

avatar
New Contributor

Do you know if there is a way to specify a python virtual environment for streaming_python to use instead of it using the base python installation?

avatar
Rising Star

Thank you. Its a good simple example for me to understand.