Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pig Script Compile Error

Pig Script Compile Error

Explorer

I am new to Pig and I am getting this error when running this script.

 

A = LOAD 'tokenized_access_logs' AS (code1, ip);
B = FILTER A by code1 > 0;
C = GROUP B BY code1;
D = FOREACH A GENERATE code1, COUNT(B);
STORE D INTO 'output';

 

I am pasting the output from the log file.  It looks like there is a parse error.  I would appreciate if someone could tell me where the mistake is.

 

Mark

 

 

Apache Pig version 0.12.0-cdh5.4.2 (rexported)
compiled May 19 2015, 17:03:41

Run pig script using PigRunner.run() for Pig version 0.8+
2015-11-04 13:52:42,293 [uber-SubtaskRunner] INFO org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.4.2 (rexported) compiled May 19 2015, 17:03:41
2015-11-04 13:52:42,294 [uber-SubtaskRunner] INFO org.apache.pig.Main - Logging error messages to: /opt/data/yarn/nm/usercache/admin/appcache/application_1445037096982_0085/container_1445037096982_00...
2015-11-04 13:52:42,425 [uber-SubtaskRunner] INFO org.apache.pig.impl.util.Utils - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://cloudera1:8020
2015-11-04 13:52:42,536 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: cloudera1:8032
2015-11-04 13:52:42,549 [uber-SubtaskRunner] WARN org.apache.pig.PigServer - Empty string specified for jar path
2015-11-04 13:52:43,145 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
Failed to parse: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
        at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1688)
        at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1421)
        at org.apache.pig.PigServer.parseAndBuild(PigServer.java:354)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:379)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:478)
        at org.apache.pig.PigRunner.run(PigRunner.java:49)
        at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
        at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
        at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
        at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10942)
        at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
        at org.apache.pig.parser.LogicalPlanGenerator.real_arg(LogicalPlanGenerator.java:9612)
        at org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:9343)
        at org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:11051)
        at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10810)
        at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
        at org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:7488)
        at org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:17590)
        at org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:15982)
        at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15849)
        at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
        at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
        at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
        at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
        ... 32 more
2015-11-04 13:52:43,148 [uber-SubtaskRunner] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar

Apache Pig version 0.12.0-cdh5.4.2 (rexported) 
compiled May 19 2015, 17:03:41

Run pig script using PigRunner.run() for Pig version 0.8+
2015-11-04 13:52:42,293 [uber-SubtaskRunner] INFO org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.4.2 (rexported) compiled May 19 2015, 17:03:41
2015-11-04 13:52:42,294 [uber-SubtaskRunner] INFO org.apache.pig.Main - Logging error messages to: /opt/data/yarn/nm/usercache/admin/appcache/application_1445037096982_0085/container_1445037096982_00...
2015-11-04 13:52:42,425 [uber-SubtaskRunner] INFO org.apache.pig.impl.util.Utils - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://cloudera1:8020
2015-11-04 13:52:42,536 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: cloudera1:8032
2015-11-04 13:52:42,549 [uber-SubtaskRunner] WARN org.apache.pig.PigServer - Empty string specified for jar path
2015-11-04 13:52:43,145 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
Failed to parse: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1688)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1421)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:354)
at org.apache.pig.PigServer.executeBatch(PigServer.java:379)
at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10942)
at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
at org.apache.pig.parser.LogicalPlanGenerator.real_arg(LogicalPlanGenerator.java:9612)
at org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:9343)
at org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:11051)
at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10810)
at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
at org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:7488)
at org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:17590)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:15982)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15849)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 32 more
2015-11-04 13:52:43,148 [uber-SubtaskRunner] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
2 REPLIES 2

Re: Pig Script Compile Error

Explorer

I ran another simple query

 

log = LOAD 'tokenized_access_logs’AS (code1, ip, url);
grpd = GROUP log BY code1;
cntd = FOREACH grpd GENERATE group, COUNT(log);
STORE cntd INTO 'output';

 

I got an error but no logs were generated.  I do see this error when I click on "ERROR'

 

  Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]

 

Can someone tell me why I get this error?  It looks like there are some classes missing.  I would appreciate your help.

 

Mark

Re: Pig Script Compile Error

Explorer

Please close this case as I dont have access to Amazon Cloud any more.  I appreciate Cloudera for providing me free access to the tools.

 

Mark