Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pig Script Compile Error

Highlighted

Pig Script Compile Error

Explorer

I am new to Pig and I am getting this error when running this script.

 

A = LOAD 'tokenized_access_logs' AS (code1, ip);
B = FILTER A by code1 > 0;
C = GROUP B BY code1;
D = FOREACH A GENERATE code1, COUNT(B);
STORE D INTO 'output';

 

I am pasting the output from the log file.  It looks like there is a parse error.  I would appreciate if someone could tell me where the mistake is.

 

Mark

 

 

Apache Pig version 0.12.0-cdh5.4.2 (rexported)
compiled May 19 2015, 17:03:41

Run pig script using PigRunner.run() for Pig version 0.8+
2015-11-04 13:52:42,293 [uber-SubtaskRunner] INFO org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.4.2 (rexported) compiled May 19 2015, 17:03:41
2015-11-04 13:52:42,294 [uber-SubtaskRunner] INFO org.apache.pig.Main - Logging error messages to: /opt/data/yarn/nm/usercache/admin/appcache/application_1445037096982_0085/container_1445037096982_00...
2015-11-04 13:52:42,425 [uber-SubtaskRunner] INFO org.apache.pig.impl.util.Utils - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://cloudera1:8020
2015-11-04 13:52:42,536 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: cloudera1:8032
2015-11-04 13:52:42,549 [uber-SubtaskRunner] WARN org.apache.pig.PigServer - Empty string specified for jar path
2015-11-04 13:52:43,145 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
Failed to parse: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
        at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1688)
        at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1421)
        at org.apache.pig.PigServer.parseAndBuild(PigServer.java:354)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:379)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at org.apache.pig.Main.run(Main.java:478)
        at org.apache.pig.PigRunner.run(PigRunner.java:49)
        at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
        at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
        at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
        at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10942)
        at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
        at org.apache.pig.parser.LogicalPlanGenerator.real_arg(LogicalPlanGenerator.java:9612)
        at org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:9343)
        at org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:11051)
        at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10810)
        at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
        at org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:7488)
        at org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:17590)
        at org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:15982)
        at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15849)
        at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
        at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
        at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
        at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
        at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
        ... 32 more
2015-11-04 13:52:43,148 [uber-SubtaskRunner] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar

Apache Pig version 0.12.0-cdh5.4.2 (rexported) 
compiled May 19 2015, 17:03:41

Run pig script using PigRunner.run() for Pig version 0.8+
2015-11-04 13:52:42,293 [uber-SubtaskRunner] INFO org.apache.pig.Main - Apache Pig version 0.12.0-cdh5.4.2 (rexported) compiled May 19 2015, 17:03:41
2015-11-04 13:52:42,294 [uber-SubtaskRunner] INFO org.apache.pig.Main - Logging error messages to: /opt/data/yarn/nm/usercache/admin/appcache/application_1445037096982_0085/container_1445037096982_00...
2015-11-04 13:52:42,425 [uber-SubtaskRunner] INFO org.apache.pig.impl.util.Utils - Default bootup file /var/lib/hadoop-yarn/.pigbootup not found
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-11-04 13:52:42,530 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://cloudera1:8020
2015-11-04 13:52:42,536 [uber-SubtaskRunner] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: cloudera1:8032
2015-11-04 13:52:42,549 [uber-SubtaskRunner] WARN org.apache.pig.PigServer - Empty string specified for jar path
2015-11-04 13:52:43,145 [uber-SubtaskRunner] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
Failed to parse: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1688)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1421)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:354)
at org.apache.pig.PigServer.executeBatch(PigServer.java:379)
at org.apache.pig.PigServer.executeBatch(PigServer.java:365)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:478)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:285)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:228)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:46)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10942)
at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
at org.apache.pig.parser.LogicalPlanGenerator.real_arg(LogicalPlanGenerator.java:9612)
at org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:9343)
at org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:11051)
at org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:10810)
at org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:10159)
at org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:7488)
at org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:17590)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:15982)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15849)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 32 more
2015-11-04 13:52:43,148 [uber-SubtaskRunner] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:
<file script.pig, line 4, column 36> Invalid scalar projection: B : A column needs to be projected from a relation for it to be used as a scalar
2 REPLIES 2

Re: Pig Script Compile Error

Explorer

I ran another simple query

 

log = LOAD 'tokenized_access_logs’AS (code1, ip, url);
grpd = GROUP log BY code1;
cntd = FOREACH grpd GENERATE group, COUNT(log);
STORE cntd INTO 'output';

 

I got an error but no logs were generated.  I do see this error when I click on "ERROR'

 

  Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]

 

Can someone tell me why I get this error?  It looks like there are some classes missing.  I would appreciate your help.

 

Mark

Re: Pig Script Compile Error

Explorer

Please close this case as I dont have access to Amazon Cloud any more.  I appreciate Cloudera for providing me free access to the tools.

 

Mark