I'll try the local build on one of the datanodes, that shouldn't be a problem for what I'm testing. It's the full dataset, but the original data was actually userID/prodDesc/weight ... I was informed by our security team that I could send the data if I changed the prodDesc to prodID, since it's pretty meaningless without lookup tables. So the Item variable went from a string when I was testing it, to a numeric; perhaps that's why I didn't see the same error. So I'm wondering if the problem is only seen if the Item variable is a string ... easy way to test it would be to hash the prodID, which would give an alpha numeric string, similar in format to the original prodDesc. I can hash the data and re-upload it, or you can run this little bit of python: #!/usr/bin/python
import csv,hashlib,sys,os,string
INPUT_FILE = csv.reader(open("cloudera_data.csv","rb"), delimiter=",")
OUTPUT_FILE = csv.writer(open("output.csv","wb"), delimiter=",")
for data_lines in INPUT_FILE:
data_lines[1] = string.upper(hashlib.sha1(string.strip(str(data_lines[1]),chars="\n")).hexdigest())
OUTPUT_FILE.writerow(data_lines)
sys.exit(0)
... View more
