Created 05-20-2016 02:00 PM
Hello,
I have Tez 0.7.0 and HDP 2.4.0.0.169. I use pig latin quite a lot in Mapreduce -mode. Today I tried Tez mode (From my java code .. ) everything seems to work fine, but in same pig latin script in Mapreduce stores alias Ok, but in tez -mode I get "ERROR 1002: Unable to store alias secondorder". I checked the script and nothing wrong there. Here is my script: (When I check path tebs/results/Ravintolamyynnintop20 from HDFS, its there stored ok .. But still I get this error Unable to store alias secondorder and show stops .. ) what could be wrong ? Some version conflict with my Java code vs. HDP ?
splittedII = load 'tebs/data/currentmon*.*' using PigStorage(';') as (id:chararray,fu:chararray,fa:chararray,myynti:chararray); splittedI = FILTER splittedII BY NOT($3 MATCHES '.*Ko.*') AND NOT($2 MATCHES '.*Yht.*') AND NOT($1 MATCHES '.*Yht.*'); onlyrestaurants = FILTER splittedI BY ($1 MATCHES '.*23021 RUOKA.*') OR ($1 MATCHES '.*23022 SUOLAINEN.*') OR ($1 MATCHES '.*23023 MAKEA.*') OR ($1 MATCHES '.*23024 PIKARUOKA.$ partly = foreach onlyrestaurants generate $1,$2,REPLACE(myynti, ',','.'); store partly into 'teb/tempten'; partly = load 'teb/tempten' using PigStorage('\t') as (id:chararray,fu:chararray,myynti:double); grpded = group partly by ($0,$1); summed = foreach grpded generate FLATTEN(group) AS (id,fu),SUM(partly.$2); ordered = order summed by $2 DESC; grp = group ordered by $0; top10 = foreach grp { sorted = order ordered by $2 desc; top = limit sorted 20; total = SUM(ordered.$2); generate group,FLATTEN(top),FLATTEN(total); }; secondorder = ORDER top10 by $4 DESC; store secondorder into 'tebs/results/Ravintolamyynnintop20';
Created 05-22-2016 10:21 AM
I found solution: Problem was about versions: I was using Tez 0.7.0 from Maven central, and hadoop 2.7.1 .. When I used 0.7.1 version from Tez this STORE -problem is gone. So problem solved.
Created 05-20-2016 04:10 PM
@petri koskican you share Pig job Logs?
Created 05-20-2016 08:25 PM
Hello! Thanks for the quick reply. Here comes log file for this job: http://pastebin.ca/3605448 Don't mind about the lines after "starting make visualisation" .. They fail for reason in this case. I am stunned, because that script stores one time temp -file, but when it comes to second time .. unable to store .. I hope you can help me with this one. @Pradeep Bhadani
Created 05-22-2016 10:21 AM
I found solution: Problem was about versions: I was using Tez 0.7.0 from Maven central, and hadoop 2.7.1 .. When I used 0.7.1 version from Tez this STORE -problem is gone. So problem solved.