@Hugo FelixHere's a test, that should help to determine if your syntax is off or if your environment is misconfigured.
First, create a test Hive table and populate it with data:
CREATE TABLE IF NOT EXISTS testtable
(id string, text string)
STORED AS ORC;
INSERT INTO TABLE testtable VALUES
('1111', 'The service was great, and the agent was very helpful'),
('2222', 'I enjoyed the event but the food was terrible'),
('3333', 'Unhappy with the organization of the event')
Then create a file called "my_py_udf.py" (as shown below). It can be placed anywhere, but in my example I placed it at /tmp/my_py_udf.py.
import sys
for line in sys.stdin:
id, text = line.replace('\n',' ').split('\t')
positive = set(["love", "good", "great", "happy", "cool", "best", "awesome", "nice", "helpful", "enjoyed"])
negative = set(["hate", "bad", "stupid", "terrible", "unhappy"])
words = text.split()
word_count = len(words)
positive_matches = [1 for word in words if word in positive]
negative_matches = [-1 for word in words if word in negative]
st = sum(positive_matches) + sum(negative_matches)
if st > 0:
print '\t'.join([text, 'positive', str(word_count)])
elif st < 0:
print '\t'.join([text, 'negative', str(word_count)])
else:
print '\t'.join([text, 'neutral', str(word_count)])
Then from within Hive, execute the following commands:
ADD FILE my_py_udf.py;
SELECT
TRANSFORM (id, text)
USING 'python my_py_udf.py'
AS (text, sentiment, word_count)
FROM testtable;
Your resulting output should look like this:
The service was great, and the agent was very helpful positive 10
I enjoyed the event but the food was terrible neutral 9
Unhappy with the organization of the event neutral 7
