Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 06-02-2016 09:38 PM
This is what Ambari says:
WARNING: Use "yarn jar" to launch YARN applications. 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/06/02 21:12:43 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Apache Pig version 0.15.0.2.4.0.0-169 (rexported) compiled Feb 10 2016, 07:50:04 2016-06-02 21:12:43,603 [main] INFO org.apache.pig.Main - Logging error messages to: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:44,356 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/yarn/.pigbootup not found 2016-06-02 21:12:44,477 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020 2016-06-02 21:12:45,021 [main] INFO org.apache.pig.PigServer - Pig Script ID for the session: PIG-script.pig-b1da2d85-2845-4a3f-b59b-b2e367f74c1c 2016-06-02 21:12:45,420 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 2016-06-02 21:12:45,646 [main] INFO org.apache.pig.backend.hadoop.ATSService - Created ATS Hook 2016-06-02 21:12:46,138 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Failed to parse: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199) at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1776) at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1484) at org.apache.pig.PigServer.parseAndBuild(PigServer.java:428) at org.apache.pig.PigServer.executeBatch(PigServer.java:453) at org.apache.pig.PigServer.executeBatch(PigServer.java:439) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:502) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:897) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 17 more Caused by: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:885) ... 23 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/NoSuchObjectException at org.apache.hive.hcatalog.pig.HCatLoader.<init>(HCatLoader.java:71) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:383) at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:742) ... 24 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 31 more 2016-06-02 21:12:46,148 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse: <file script.pig, line 4, column 4> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'org.apache.hive.hcatalog.pig.HCatLoader' with arguments 'null' Details at logfile: /hadoop/yarn/local/usercache/admin/appcache/application_1464899556124_0006/container_e16_1464899556124_0006_01_000002/pig_1464901963602.log 2016-06-02 21:12:46,167 [main] INFO org.apache.pig.Main - Pig script completed in 2 seconds and 722 milliseconds (2722 ms)
Created on 06-08-2016 07:45 PM - edited 08-19-2019 03:47 AM
Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.
Created 06-02-2016 10:02 PM
Are you able to run same pig script through command line?
Also looks like below jar either missing or have multiple versions while runing pig.
/usr/hdp/current/hive-client/lib/hive-metastore-<version>.jar
Created 06-02-2016 10:02 PM
I tried a couple of things. Ran the job via command line using this syntax "pig -x tez -useHCatalog -f script.pig" ...this works fine. The problem is, I need to find its equivalent to make it run via Pig View in Ambari.
Created 06-03-2016 07:50 PM
I am able to reproduce this via CLI as well. For a refresher, working on a Sandbox downloaded from Hortonworks site, also, when I manually specify the HADOOP_CLASSPATH and HADOOP_CLIENT_OPTS via command line, I get it to work. I got the classpath and hadoop client opts using "pig -printCmdDebug -x tez -useHCatalog" there are indeed some additional jar's that are included in the hadoop class path and client opts.
Created 06-07-2016 11:29 PM
same here with the latest sandbox, won't work from Ambari
Created on 06-08-2016 07:45 PM - edited 08-19-2019 03:47 AM
Have you tried adding "-useHCatalog" as a pig argument? On the pig script, there is an option in the lower right to add an argument. This worked for me. Without this option, I was unable to run an pig scripts in Ambari that required HCat based jars.
Created 06-08-2016 08:10 PM
Note this tutorial covers it in step 3.3: http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#...
Created 06-30-2016 02:16 PM
m also facing same issue can someone help on this @Srai.
M not able to run pig queries using hcatalog
Created 06-30-2016 02:59 PM
hey Abhijeet, not sure if you have tried this.
"pig -printCmdDebug -x tez -useHcatalog"
Let me know if this does not work for you.
Created 06-30-2016 03:28 PM
Hey Srai thanks for replying m newbie to this hadoop world.
Can you please tell me where i need to execute this command.
I tried in both directly from shell :
[root@sandbox ~]# pig -printCmdDebug -x tez -useHcatalog;
and in pig using mapreduce mode
grunt> pig -printCmdDebug -x tez -useHcatalog;
in both case its not working @srai.